OpenPI Robot Policy WebSocket API¶
Use WS /v1/realtime/robot/openpi for low-latency robot-policy inference with OpenPI-compatible clients. A persistent WebSocket carries observations to the loaded policy model and returns action arrays.
Despite the /realtime prefix, this endpoint does not use the OpenAI Realtime event schema. It uses binary MessagePack frames with NumPy extensions.
Availability¶
The endpoint is available only when the loaded diffusion policy configuration contains policy_server_config. The server sends that model-specific configuration to the client as the first binary frame.
For example, DreamZero declares:
policy_server_config:
image_resolution: [180, 320]
n_external_cameras: 2
needs_wrist_camera: true
needs_stereo_camera: false
needs_session_id: true
action_space: joint_position
Without this configuration, the WebSocket returns Robot policy not available and closes.
Multi-stage policies¶
A policy can run as several stages, for example an autoregressive stage that reasons before a diffusion stage that acts. The diffusion stage declares policy_server_config as above and receives the observation; every other stage gets its deploy default_sampling_params, and an observation seed applies to all stages. The reply is the actions of the stage that returns them.
An inference message can ask for several samples with "sampling_params": {"n": <int>}, which sets the first stage's n. It needs a policy whose first stage sends one KV cache per sample (omni_kv_config["per_sample_kv"], for example Alpamayo 2 Super); the reply then stacks one action chunk per sample along a new leading axis, and sample i uses seed seed + i. Other policies reject n > 1.
When the first stage needs a prompt built from the observation (token ids, images), the diffusion stage's model_config also names openpi_prompt_builder, a class following the OpenPIPromptBuilder protocol in vllm_omni/entrypoints/openpi/serving.py. It is constructed with the engine's vLLM ModelConfig and the policy_server_config values, and its build_prompt(obs) returns the first stage's prompt. It is not sent to clients. In the deploy config:
stages:
- stage_id: 1 # the diffusion stage
model_config:
policy_server_config:
action_horizon: 64
# ...
openpi_prompt_builder: my_package.policy.MyOpenPIPrompt
DreamZero Quick Start¶
Start the bundled DreamZero deployment:
vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8091 \
--served-model-name dreamzero-droid \
--deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \
--enforce-eager --disable-log-stats
Install the optional client dependencies, download the sample camera inputs, and run the client:
pip install openpi-client websockets opencv-python
hf download YangshenDeng/vllm-omni-dreamzero-assets \
--repo-type dataset \
--local-dir outputs/dreamzero/assets
python examples/online_serving/dreamzero/openpi_client.py \
--host 127.0.0.1 \
--port 8091 \
--video-dir outputs/dreamzero/assets
Protocol¶
The connection is request-response after an initial server handshake:
connect
<- msgpack(policy_server_config)
infer
-> msgpack({"endpoint": "infer", "session_id": "...", "seed": <optional int>, ...observation})
<- msgpack(ndarray | dict[str, ndarray])
reset
-> msgpack({"endpoint": "reset"})
<- msgpack({"status": "reset successful"})
If endpoint is omitted, the server treats the message as infer. Each inference message produces one action response after the engine request completes; action tokens or intermediate tensors are not streamed.
NumPy arrays use the marker format implemented by openpi-client. The server also accepts the legacy vLLM NumPy marker representation on input. JSON text frames are not observation messages.
Sessions and Reset¶
- The API layer's current-session and first-call counters are scoped to each WebSocket connection.
session_ididentifies model-side state across observations on that connection. If omitted, it defaults todefault.- The first inference for a session is sent to the model with
reset=true. - Changing
session_id, or sending theresetcommand, causes the next inference to start withreset=true. - The policy pipeline owns observation transforms and persistent model state, normally keyed by
session_id; the API layer forwards the raw observation dictionary. - An optional integer
seedin an inference message becomes the engine request'ssampling_params.seed. A policy whose sampling consumes the request generator (GR00T-N1.7) returns the same action chunk for the same observation and seed. If omitted, the engine assigns a random per-request seed, so unseeded inferences are independent of each other. The API layer consumesseed; it is not forwarded inside the observation.
Limits and Errors¶
- Maximum inbound payload size is 64 MiB.
- The server closes an idle connection after 30 seconds.
- Invalid binary input returns a MessagePack
{"type":"error","message":"Invalid request payload"}response. - Inference failures return a generic
Internal inference errorwithout exposing an internal traceback. - Unsupported NumPy object, structured, and complex dtypes are rejected.
- A configured
openpi_prompt_builderthat cannot be imported fails server startup with an error naming the setting.
Observation keys, camera layout, state tensors, and action shapes are defined by the loaded policy rather than by this transport. Read the handshake before constructing observations. See the DreamZero example for a complete OpenPI client and DROID simulation loop.