Streaming Video Output WebSocket API¶
Use WS /v1/realtime/video to receive fragmented MP4 chunks while a compatible diffusion video model is generating. This endpoint streams generated video output; it is unrelated to WS /v1/video/chat/stream, which streams live video input into a conversational model.
Quick Start¶
Start a model that supports step execution and enable diffusion streaming output:
Install the client dependency and run the provided client:
pip install av websockets
python examples/online_serving/streaming_video_generation/streaming_video_client.py \
--host 127.0.0.1 \
--port 8091 \
--model BestWishYsh/Helios-Distilled \
--prompt "A serene lakeside sunrise with mist over the water" \
--output output.mp4
Startup Warmup¶
Use --video-output-warmup REQUEST_JSON to complete a short generation and fragmented-MP4 encode before the HTTP server starts. This moves first-request initialization into startup: /health and user routes become available only after the warmup succeeds. It increases startup time and does not promise higher steady-state generation throughput. The option is disabled when omitted.
The JSON file uses the session.start fields below. Supply an explicit num_frames (preferably one native generation chunk), the deployment's output dimensions, and any model-required image or camera inputs. Omit playback pacing: the internal consumer drains output immediately and has no player to send feedback. Model sampling defaults are inherited unless the request overrides them. Use a synthetic image and prompt rather than private user content.
For example, create a one-chunk request for the LingBot-World stepwise deployment:
import json
from io import BytesIO
from pathlib import Path
import pybase64
from PIL import Image
# Match ar_diffusion_width/height in the deployment configuration.
width, height = 832, 480
image = Image.new("RGB", (width, height), (80, 120, 160))
image.paste((160, 120, 80), (width // 2, 0, width, height))
buffer = BytesIO()
image.save(buffer, format="PNG")
request = {
"type": "session.start",
"prompt": "A quiet landscape in daylight.",
"image_reference": {
"image_url": "data:image/png;base64," + pybase64.b64encode(buffer.getvalue()).decode("ascii")
},
"width": width,
"height": height,
"num_frames": 9,
"fps": 12,
"seed": 42,
"extra_params": {"camera_action_script": [[["w"], ["w"], ["w"]]]},
}
Path("video-warmup.json").write_text(json.dumps(request), encoding="utf-8")
Then add the warmup file to the server command:
vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers --omni \
--deploy-config vllm_omni/deploy/lingbot_world_v2_stepwise.yaml \
--disable-websocket-compression \
--video-output-warmup video-warmup.json \
--video-output-warmup-timeout 180
The generation/encoding deadline defaults to 180 seconds and must be finite and positive. Invalid requests, missing encoded video, generation errors, and timeouts fail startup instead of advertising readiness. Failure cleanup also uses the normal bounded request-abort timeout. Adjust the deployment's startup probe budget to include model loading and warmup. The warmup exercises generation and the encoder directly; it does not open a WebSocket or warm network transport. One request warms the selected execution path, not every shape, LoRA or replica.
Client Frame Diagnostics¶
The generic streaming client counts decoded frames on a background worker so accounting does not hold up reception or scheduled controls. Its chunk log fields have distinct meanings:
framesis the server-declarednum_frames, orunknownwhen absent.total_framesis an asynchronous snapshot of frames actually decoded so far. It may lag reception; differences between snapshots are not per-chunk frame counts or instantaneous generation FPS.
The final [decode] line follows end-of-input draining and includes decodable trailer frames. For interrupted streams it counts only decodable received media. Its elapsed time includes reception and draining, rather than model inference alone. Accounting failure, a 64 MiB pending-input overflow, cancellation during the drain, or a five-second worker shutdown timeout reports total_frames=unknown with a status reason. These accounting limits do not bound the whole-session bytes retained for saving or change the final save path.
See the streaming video example README for commands, buffering, and cancellation details.
Protocol¶
Control events are JSON text frames. Each video.chunk_metadata event is immediately followed by one binary WebSocket frame containing fragmented MP4 bytes. If an encoding call emits no bytes yet, the server still sends its metadata followed by an empty binary frame (byte_length: 0). This preserves source counts and interaction events. Empty binary frames add no media to the stream and do not indicate completion; wait for session.done.
An append-only decoder can produce a generation chunk with no stable frames yet. Its metadata is still sent with num_frames=0 and byte_length=0, followed by an empty binary frame. This preserves chunk positions and interaction acknowledgements while the decoder holds future-dependent pixels.
Models may also provide frame_start (the cumulative visible output position), generated_raw_frames (decoder positions before segment padding is omitted), generated_frames (available visible frames), and withheld_frames (available frames waiting for future context). Normal completion flushes withheld frames; cancellation discards them.
| Direction | Event or frame | Purpose |
|---|---|---|
| Client to server | session.start | Starts generation with a prompt and video parameters |
| Server to client | video.start | Confirms the request ID, format, and accepted configuration |
| Server to client | video.chunk_metadata | Describes the next binary media frame |
| Server to client | Binary frame | Carries one fragmented MP4 chunk |
| Client to server | session.interaction | Queues an optional interaction (e.g., prompt. camera action) during generation |
| Server to client | session.interaction.queued | Confirms that an interaction was queued |
| Client to server | session.ping | Refreshes the stall clock |
| Server to client | session.pong | Acknowledges a ping |
| Client to server | session.stop | Cancels the active generation; the resulting session.done reports stop_latency_s |
| Server to client | session.done | Reports completion or cancellation and the chunk count |
| Server to client | error | Reports validation, generation, or timeout failures |
Start a minimal session with:
{
"type": "session.start",
"model": "BestWishYsh/Helios-Distilled",
"prompt": "A serene lakeside sunrise",
"format": "m4s",
"width": 640,
"height": 384,
"fps": 16,
"num_frames": 99
}
Only m4s output is currently accepted. Concatenate the binary frames in receive order, then remux the fragmented stream if the target player requires a progressive MP4 file.
Audio and chunk boundaries¶
A generation result containing audio must contain exactly one video; multiple videos with one audio payload are rejected before encoding any of that result. Video-only results may contain multiple videos. Their generation chunk index, interaction event IDs, pacing fields and output chunk index are reported only with the first video. Each video retains its own submitted frame count and emitted byte length.
If the first video chunk includes audio, the server creates an AAC track. Every later video chunk must then include nonempty audio. An explicit sample rate must be a positive integer and must match the track's initial rate; invalid chunks are rejected before advancing either media timeline. Omitting a later sample rate keeps the track's rate. If no initial rate is provided, the server uses 24,000 Hz. Audio cannot be added after a video-only MP4 header has been emitted.
The binary frames form one continuous fMP4 stream, rather than independently decodable files. In video.chunk_metadata, byte_length describes the following binary frame. num_frames, num_audio_samples, generation_chunk_index and interaction event IDs describe the source media submitted during that encoding call; they do not count the media packets already present in those bytes. audio_sample_rate reports the rate used for the audio track.
The audio/video muxer uses a 100 ms fragment-duration threshold and retains an incomplete final fragment. AAC also buffers incomplete 1024-sample frames and encoder delay. A chunk's tail can therefore arrive with the next chunk or the final trailer. This is buffering across chunk boundaries, not an additional 100 ms of timestamp drift accumulated on every chunk. A trailer reports zero submitted frames but can contain buffered media. Clients must append bytes in receive order and use the MP4 timestamps for playback and playback feedback, rather than treating the source counts or event IDs as exact playback markers. When progressive-MP4 finalization fails, saving retains the received fMP4 bytes.
AAC encoder priming is represented by the MP4 edit list, rather than by dropping generated samples. The initialization header is delayed until FFmpeg has the initial timestamps. Encoded AAC packets can start at negative timestamps; presented audio and video start at zero after the player's priming adjustment. Clients must honor the edit list (elst) in the fMP4 initialization segment, including when using Media Source Extensions (MSE). Ignoring it can delay audio by one AAC frame (1024 samples, or 32 ms at 32 kHz). Playback checks here use PyAV/FFmpeg; browser MSE playback has not been validated.
Playback Backpressure¶
A compatible single-stage interactive pipeline can pace generation against a client playhead. Enable step execution, streaming_output=True and max_num_seqs=1, then include these fields in session.start:
{
"streaming_pacing": {
"enabled": true,
"mode": "client_playhead",
"max_lead_ms": 1500,
"max_output_bytes": 67108864,
"idle_timeout_s": 30
}
}
This is a request fragment; include the model's normal prompt, image and video parameters. Omitting streaming_pacing or setting enabled to false keeps unpaced execution. streaming_buffer_seconds remains a compatibility alias for existing clients. The ordinary /v1/videos endpoint does not accept playback pacing.
While playing, repeatedly send the actual cumulative media playhead:
The engine starts the next chunk only if generated duration + next chunk duration - played duration fits the configured window. The window must fit a whole chunk. Old feedback is ignored and future feedback is clamped to already generated media. Paused playback blocks advancement; pings and duplicate playback positions do not prevent the consumption idle timeout.
A paced stream retains at most one generation chunk until encoding and sending finish. The byte limit checks decoded payloads and encoded fragments; it does not cap model workspace, transient decode allocations or process RSS. Oversize output, idle timeout, Stop and disconnect release request state without waiting for another inference request. Other requests retain the single-stream queueing behavior.
Chunk metadata includes pacing.lead_seconds, cumulative pacing.wait_seconds, pacing.retained_bytes and output_chunk_index. AsyncOmni.generate consumers release a chunk by advancing their async iterator; close or abort an abandoned iterator. Controls and playback feedback remain receivable while media output is blocked, through bounded control queues.
For interaction timing, queue limits and cancellation semantics, see the streaming video example protocol.
Mid-Generation Interaction¶
Models that implement interaction updates can change the active prompt:
{
"type": "session.interaction",
"interaction": {
"event_id": "sunlight-update",
"event": {"prompt": "Sunlight breaks through the mist"},
"transition_chunks": 2
}
}
Models that register the camera modality also accept camera updates on event.multi_modal_data.camera; the payload is model-specific. LingBot-World takes structural SE3 payloads. Coordinates follow Unity semantics (+X right, +Y up, +Z forward; same as LingBot's native camera frame), and WASD key tokens are not admitted:
{
"type": "session.interaction",
"interaction": {
"event_id": "forward-1",
"event": {
"multi_modal_data": {
"camera": {
"mode": "velocity",
"data": {
"translation": [0.0, 0.0, 0.05],
"rotation": [0.0, 0.0, 0.0, 1.0]
}
}
}
}
}
}
mode is target (lerp to an absolute pose over transition_chunks) or velocity (hold a per-frame SE3 delta until replaced). Prompt and camera may be combined in one interaction event.
ABot-World instead conditions on W/A/S/D/I/J/K/L key actions, sent in state or script mode, and registers only the camera modality; see the ABot-World recipe.
The example client's --prompt-updates and --camera-updates schedules assign a distinct event_id to each entry within the session, including entries with equal timestamps or timestamps less than a millisecond apart. These IDs are opaque identifiers; at still determines the scheduled send time. The server continues to reject repeated IDs with event_id_conflict.
This feature is pipeline-dependent. An enabled endpoint does not imply that the loaded model supports midway prompt or camera updates.
session.interaction.queued confirms only that the server accepted the message. If the model rejects the interaction later, for example because the modality is unsupported or the payload is invalid, the server sends an error with "code": "interaction_failed" and the same event_id. Generation keeps streaming, and the event_id may be reused.
The default start timeout is 10 seconds and the server reports a stall after about 60 seconds without generation progress or a session.ping. See the streaming video generation example for prompt/camera-update scheduling and browser playback.