Handle binary WebSocket frames

Connect with binary frames enabled and decode the OMNI frames the facade sends for streaming audio. Binary frames carry the raw bytes, so the client skips base64 decoding.

Prerequisites

A voice agent: the facade sends binary frames only for duplex audio output.
A WebSocket client that can receive binary messages.

Connect with binary frames enabled

Add binary=true to the connection URL and read WebSocket messages as ArrayBuffer:

const ws = new WebSocket('ws://localhost:8080/ws?agent=my-agent&binary=true');
ws.binaryType = 'arraybuffer';

The connected message reports connected.capabilities.binary_frames: true when the facade accepted the request. A client that omits binary=true receives duplex audio as JSON media_chunk messages with base64 data.

Decode a frame

Every binary message starts with a 32-byte header. Read the integers big-endian, then the JSON metadata, then the payload:

ws.onmessage = (event) => {
  if (!(event.data instanceof ArrayBuffer)) {
    handleJson(JSON.parse(event.data));
    return;
  }
  const view = new DataView(event.data);
  const magic = new TextDecoder().decode(new Uint8Array(event.data, 0, 4));
  if (magic !== 'OMNI') return;

  const flags = view.getUint8(5);
  const metadataLen = view.getUint32(8, false);
  const sequence = view.getUint32(16, false);
  const isLast = (flags & 0x04) !== 0;

  const metadata = JSON.parse(
    new TextDecoder().decode(new Uint8Array(event.data, 32, metadataLen)));
  const payload = event.data.slice(32 + metadataLen);

  handleMediaChunk(metadata, sequence, isLast, payload);
};

The offsets and flag bits are in the WebSocket protocol reference.

Reassemble chunked payloads

The facade splits a payload larger than 1 MB into 64 KB frames and sets the chunked flag (flags & 0x02) on each. For these frames:

Read total_chunks from the metadata.
Treat sequence as the zero-based chunk index.
Concatenate the payloads in sequence order until you have total_chunks of them.

Smaller payloads arrive in one frame.

Send audio to the agent

A client sends microphone audio as binary frames of the same layout, with message type 1. Set the last-frame flag (0x04) only on the final frame of the call, because it ends the audio session. The first frame carries codec, sample_rate and channels in its metadata (defaults: pcm, 16000, 1). The runtime can answer with a session_config message, which the facade relays, giving the format to capture at.

Verify

Log sequence and the payload length for each frame while the agent speaks. The connected message must report binary_frames: true, and frames must arrive as ArrayBuffer messages rather than JSON.