Realtime Audio WebSocket API¶

Use WS /v1/realtime to stream PCM audio into a compatible omni model and receive incremental transcription and synthesized response audio. This is a turn-based streaming API: it starts one generation over an audio stream and finishes after the client closes that input stream.

For a model that listens while it is already speaking, use the Full-Duplex API instead.

Quick Start¶

Start a compatible model. The currently documented example uses Qwen3-Omni:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file:

python examples/online_serving/qwen3_omni/openai_realtime_client.py \
  --url ws://localhost:8091/v1/realtime \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --input-wav input_16k_mono.wav \
  --output-wav response.wav \
  --output-text response.txt

The client dependency is websockets.

Protocol¶

Messages in both directions are JSON text frames. Audio carried inside an event is base64-encoded raw PCM16.

Direction	Event	Purpose
Server to client	`session.created`	Confirms that the WebSocket connection is ready
Client to server	`session.update`	Selects and validates the served model
Client to server	`input_audio_buffer.commit` with `final: false`	Starts generation over the incoming stream
Client to server	`input_audio_buffer.append`	Appends base64 PCM16 audio
Client to server	`input_audio_buffer.commit` with `final: true`	Marks the end of input
Server to client	`transcription.delta`	Carries incremental response text
Server to client	`transcription.done`	Carries final text and token usage
Server to client	`response.output_audio.delta`	Carries incremental PCM16 response audio
Server to client	`response.output_audio.done`	Marks the end of response audio
Server to client	`error`	Reports an invalid event, model, or audio payload

A minimal client sends events in this order:

{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"}
{"type":"input_audio_buffer.commit","final":false}
{"type":"input_audio_buffer.append","audio":"<base64-pcm16>"}
{"type":"input_audio_buffer.commit","final":true}

The initial non-final commit intentionally starts generation before all audio has arrived. Continue sending input_audio_buffer.append events while the engine is consuming the stream.

Audio Handling¶

Input is mono PCM16 at 16 kHz for the Qwen3-Omni example.
response.output_audio.delta.audio contains base64-encoded PCM16 bytes.
Read sample_rate_hz from each audio event instead of assuming an output rate. Qwen3-Omni output is typically 24 kHz.
Concatenate audio deltas in receive order to construct the output waveform.
Set input_audio_buffer.commit.final to true after the last input chunk so the server can terminate the streaming request.

Availability and Limitations¶

The path is registered on the unified API server, but it is usable only when the loaded pipeline implements realtime audio input and produces compatible audio output. Unsupported deployments return an error event. The endpoint does not provide duplex session resume, playback acknowledgement, overlap policy, or barge-in controls.

See the Qwen3-Omni online serving example for concurrency options, chunk pacing, and per-delta audio debugging.