Videos API¶

vLLM-Omni provides an OpenAI-compatible video generation API for diffusion video models. The API supports asynchronous video jobs through /v1/videos and a synchronous benchmark-oriented endpoint through /v1/videos/sync.

Each server instance runs a single model specified at startup with vllm serve <model> --omni.

Quick Start¶

Start the Server¶

vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091

Create a Video Job¶

create_response=$(curl -s http://localhost:8091/v1/videos \
  -F "prompt=A cinematic tracking shot of a mountain lake at sunrise" \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16" \
  -F "num_inference_steps=40")

video_id=$(echo "${create_response}" | jq -r '.id')

Poll and Download¶

curl -s "http://localhost:8091/v1/videos/${video_id}" | jq .
curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4

API Reference¶

Endpoints¶

Request Parameters¶

POST /v1/videos and POST /v1/videos/sync accept multipart/form-data.

OpenAI-style fields¶

vLLM-Omni extension fields¶

Create Response¶

POST /v1/videos returns a job record:

{
  "id": "video-123",
  "status": "queued",
  "created_at": 1701234567
}

The final content is available from /v1/videos/{video_id}/content after the job status becomes completed.

Synchronous Response¶

POST /v1/videos/sync blocks until generation finishes and returns raw video bytes. It is useful for benchmarks and simple scripts that do not need job storage or polling.

Examples¶

Image-to-Video¶

curl -s http://localhost:8091/v1/videos \
  -F "prompt=animate this image with subtle camera movement" \
  -F "[email protected]" \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16"

Video-to-Video¶

For models that support video conditioning, upload the reference video with input_reference:

curl -s http://localhost:8091/v1/videos \
  -F "prompt=continue this motion with consistent subjects and lighting" \
  -F "[email protected];type=video/mp4" \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16"

You can also pass a JSON-safe video URL or data:video/...;base64,... payload through video_reference. Do not send video_reference together with input_reference or image_reference.

curl -s http://localhost:8091/v1/videos \
  -F "prompt=continue this motion with consistent subjects and lighting" \
  -F 'video_reference={"video_url":"https://example.com/input.mp4"}' \
  -F "width=1280" \
  -F "height=720" \
  -F "num_frames=80" \
  -F "fps=16"

JSON references currently support image_url/video_url; file_id references are not implemented yet. Models may expose additional V2V controls through extra_params. For example, Cosmos3 supports condition_frame_indexes_vision and condition_video_keep to select which decoded reference frames are used as clean conditioning. Cosmos3 transfer mode also accepts edge, blur, depth, seg, or wsm control hints. Each hint may specify its own control_path and control_weight. Request-level transfer options include control_guidance, control_guidance_interval, emphasize_control_in_prompt, num_video_frames_per_chunk, num_conditional_frames, show_control_condition, and show_input. Transfer uses its transfer-specific system prompt and, by default, appends a control-adherence directive to the positive prompt. It adds duration/FPS and resolution metadata to both CFG branches but does not add a negative prompt automatically. The Cosmos3 recipe includes an optional reference negative prompt and shows how to pass it. Set emphasize_control_in_prompt, use_duration_template, or use_resolution_template to false to disable the corresponding addition. negative_metadata_mode accepts same, inverse, or none and defaults to same for transfer. See the Cosmos3 recipe for complete examples.

A client that cannot place the control on the server filesystem can upload one control with control_reference and identify it with control_type. The API streams the upload to request-scoped storage, supplies its path to the model, and removes it after synchronous or asynchronous generation completes. Other options for the selected control can remain in extra_params; do not also set control or control_path there. Uploads larger than 512 MiB are rejected.

curl -s http://localhost:8091/v1/videos/sync \
  -F "prompt=Preserve the scene while following the world-state control" \
  -F "[email protected];type=video/mp4" \
  -F "[email protected];type=video/mp4" \
  -F "control_type=wsm" \
  -F 'extra_params={"wsm":{"control_weight":1.0}}' \
  -o output.mp4

HTTP redirects for image_reference.image_url follow vLLM's VLLM_MEDIA_URL_ALLOW_REDIRECTS setting. Before starting the server, set it to 1 (the default) to allow redirects or 0 to reject them. A redirect target can differ from the original host, and vLLM-Omni does not yet fully implement upstream vLLM's media URL allowlist protection. Deployments should therefore treat remote media URLs as untrusted and choose this setting as part of their URL access policy.

Speech-to-Video¶

For models that support audio-driven generation (e.g., Wan2.2-S2V), pass both an image reference and an audio reference. The audio_reference field accepts a JSON string with audio_url pointing to an HTTP(s) URL or base64 data URL.

curl -s http://localhost:8091/v1/videos \
  -F "prompt=A person singing" \
  -F 'image_reference={"image_url": "https://example.com/face.png"}' \
  -F 'audio_reference={"audio_url": "https://example.com/speech.mp3"}' \
  -F "width=832" \
  -F "height=480" \
  -F "num_inference_steps=40" \
  -F "guidance_scale=4.5" \
  -F "fps=16"

Synchronous Generation¶

curl -X POST http://localhost:8091/v1/videos/sync \
  -F "prompt=A small robot walking through a neon city" \
  -F "width=854" \
  -F "height=480" \
  -F "num_frames=80" \
  -F "fps=16" \
  -o output.mp4

Output Encoding¶

These extra_params control how the server turns decoded frames into MP4 bytes.

With preencode_mp4 enabled, each committed VAE chunk leaves the accelerator and is encoded while later chunks are still decoding, so host transfer and CPU encoding overlap the remaining decode instead of following it. The response is unchanged: the same complete MP4, byte-for-byte equivalent frames.

curl -X POST http://localhost:8091/v1/videos/sync \
  -F "prompt=A small robot walking through a neon city" \
  -F 'extra_params={"preencode_mp4": true, "video_codec_options": {"preset": "ultrafast"}}' \
  -o output.mp4

preencode_mp4 applies to the complete-MP4 response paths only. The /v1/realtime/video WebSocket endpoint rejects it, because that path already overlaps encoding through its own incremental fragmented-MP4 encoder.

Support is per model: MiniMax-H3 implements it, and other models ignore the flag and take the full-decode path.

Storage¶

Set VLLM_OMNI_SERVER_STORAGE__PATH to control where asynchronous video outputs are stored:

export VLLM_OMNI_SERVER_STORAGE__PATH=/var/tmp/vllm-omni-videos

VLLM_OMNI_STORAGE_PATH is deprecated and will be removed in a future release; use VLLM_OMNI_SERVER_STORAGE__PATH instead.

Model-Specific Examples¶

For complete text-to-video, image-to-video, and model-specific video-to-video walkthroughs, see:

• Text-to-Video
• Image-to-Video
• Speech-to-Video for Wan2.2-S2V audio-driven lip-sync generation
• Cosmos3 recipes for model-specific video-to-video examples and conditioning controls