Skip to content

OpenPI Robot Policy WebSocket API

Use WS /v1/realtime/robot/openpi for low-latency robot-policy inference with OpenPI-compatible clients. A persistent WebSocket carries observations to the loaded policy model and returns action arrays.

Despite the /realtime prefix, this endpoint does not use the OpenAI Realtime event schema. It uses binary MessagePack frames with NumPy extensions.

Availability

The endpoint is available only when the loaded diffusion policy configuration contains policy_server_config. The server sends that model-specific configuration to the client as the first binary frame.

For example, DreamZero declares:

policy_server_config:
  image_resolution: [180, 320]
  n_external_cameras: 2
  needs_wrist_camera: true
  needs_stereo_camera: false
  needs_session_id: true
  action_space: joint_position

Without this configuration, the WebSocket returns Robot policy not available and closes.

Multi-stage policies

A policy can run as several stages, for example an autoregressive stage that reasons before a diffusion stage that acts. The diffusion stage declares policy_server_config as above and receives the observation; every other stage gets its deploy default_sampling_params, and an observation seed applies to all stages. The reply is the actions of the stage that returns them.

An inference message can ask for several samples with "sampling_params": {"n": <int>}, which sets the first stage's n. It needs a policy whose first stage sends one KV cache per sample (omni_kv_config["per_sample_kv"], for example Alpamayo 2 Super); the reply then stacks one action chunk per sample along a new leading axis, and sample i uses seed seed + i. Other policies reject n > 1.

When the first stage needs a prompt built from the observation (token ids, images), the diffusion stage's model_config also names openpi_prompt_builder, a class following the OpenPIPromptBuilder protocol in vllm_omni/entrypoints/openpi/serving.py. It is constructed with the engine's vLLM ModelConfig and the policy_server_config values, and its build_prompt(obs) returns the first stage's prompt. It is not sent to clients. In the deploy config:

stages:
  - stage_id: 1  # the diffusion stage
    model_config:
      policy_server_config:
        action_horizon: 64
        # ...
      openpi_prompt_builder: my_package.policy.MyOpenPIPrompt

DreamZero Quick Start

Start the bundled DreamZero deployment:

vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8091 \
  --served-model-name dreamzero-droid \
  --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \
  --enforce-eager --disable-log-stats

Install the optional client dependencies, download the sample camera inputs, and run the client:

pip install openpi-client websockets opencv-python

hf download YangshenDeng/vllm-omni-dreamzero-assets \
  --repo-type dataset \
  --local-dir outputs/dreamzero/assets

python examples/online_serving/dreamzero/openpi_client.py \
  --host 127.0.0.1 \
  --port 8091 \
  --video-dir outputs/dreamzero/assets

Protocol

The connection is request-response after an initial server handshake:

connect
  <- msgpack(policy_server_config)

infer
  -> msgpack({"endpoint": "infer", "session_id": "...", "seed": <optional int>, ...observation})
  <- msgpack(ndarray | dict[str, ndarray])

reset
  -> msgpack({"endpoint": "reset"})
  <- msgpack({"status": "reset successful"})

If endpoint is omitted, the server treats the message as infer. Each inference message produces one action response after the engine request completes; action tokens or intermediate tensors are not streamed.

NumPy arrays use the marker format implemented by openpi-client. The server also accepts the legacy vLLM NumPy marker representation on input. JSON text frames are not observation messages.

Sessions and Reset

  • The API layer's current-session and first-call counters are scoped to each WebSocket connection.
  • session_id identifies model-side state across observations on that connection. If omitted, it defaults to default.
  • The first inference for a session is sent to the model with reset=true.
  • Changing session_id, or sending the reset command, causes the next inference to start with reset=true.
  • The policy pipeline owns observation transforms and persistent model state, normally keyed by session_id; the API layer forwards the raw observation dictionary.
  • An optional integer seed in an inference message becomes the engine request's sampling_params.seed. A policy whose sampling consumes the request generator (GR00T-N1.7) returns the same action chunk for the same observation and seed. If omitted, the engine assigns a random per-request seed, so unseeded inferences are independent of each other. The API layer consumes seed; it is not forwarded inside the observation.

Limits and Errors

  • Maximum inbound payload size is 64 MiB.
  • The server closes an idle connection after 30 seconds.
  • Invalid binary input returns a MessagePack {"type":"error","message":"Invalid request payload"} response.
  • Inference failures return a generic Internal inference error without exposing an internal traceback.
  • Unsupported NumPy object, structured, and complex dtypes are rejected.
  • A configured openpi_prompt_builder that cannot be imported fails server startup with an error naming the setting.

Observation keys, camera layout, state tensors, and action shapes are defined by the loaded policy rather than by this transport. Read the handshake before constructing observations. See the DreamZero example for a complete OpenPI client and DROID simulation loop.