Replica Data Parallelism for Video Generation¶

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/replica_data_parallel.

A runnable recipe + benchmark for replica data parallelism (replica-DP) on a video DiT. Replica-DP runs N independent diffusion engine replicas, one per GPU, and routes each request to a single replica. It scales request throughput near-linearly; it does not change single-request latency (for that, use an intra-request axis such as tensor/sequence/CFG parallelism).

This example uses Wan-AI/Wan2.2-TI2V-5B-Diffusers with one GPU per replica.

Files¶

Run¶

Start the server with the replica count you want (one GPU per replica):

# baseline
NUM_REPLICAS=1 DEVICES=0        ./run_server.sh
# 2 replicas
NUM_REPLICAS=2 DEVICES=0,1      ./run_server.sh
# 4 replicas
NUM_REPLICAS=4 DEVICES=0,1,2,3  ./run_server.sh

In another shell, drive a fixed workload and read the throughput:

python bench_replica_dp.py --url http://127.0.0.1:8098 \
    --num-requests 28 --concurrency 8 --label "replicas=4"

Re-run the server at NUM_REPLICAS=1,2,4 (same client workload each time) and compare videos/min. Pass --baseline-tpm <N=1 value> to print a scaling factor.

Isolation check¶

Identical outputs under a shared prompt+seed prove nothing -- every request would be identical regardless of isolation. Instead each request gets a distinct input (a prompt from the pool + a unique seed base_seed+i), and we check that a request's output is unchanged whether it runs alone or concurrently amid other distinct requests. Capture a per-input baseline on N=1, then verify it under N>=2 concurrency:

# 1) against NUM_REPLICAS=1 (serial), record each input's output hash
python bench_replica_dp.py --num-requests 8 --concurrency 1 \
    --write-baseline baseline.json

# 2) against NUM_REPLICAS>=2, replay the SAME inputs concurrently and verify
python bench_replica_dp.py --num-requests 8 --concurrency 8 \
    --check-baseline baseline.json

Step 2 exits non-zero if any request's output differs from its own baseline (a mismatch means cross-request state bleed or misrouting). Keep --num-requests, --base-seed, and the prompt source identical between the two runs so the input sets match.

Measured scaling¶

Wan2.2-TI2V-5B (832×480, 33 frames, 30 steps), 4× A800-80GB (NVLink), one GPU per replica:

Per-request latency stayed flat (~13–14 s) across all replica counts. Isolation is checked per-input via the baseline/verify protocol above: each request's output under N>=2 concurrency must match its own single-replica baseline.

Notes on the config surface¶

• runtime.num_replicas: N fans out N replicas; runtime.devices assigns their GPUs. With tensor_parallel_size = 1, list one GPU per replica (num_replicas * tensor_parallel_size entries, pool mode).
• Replica fan-out here comes from the deploy config, not a CLI flag. (The headless / multi-runtime launch path uses the process-local --omni-dp-size-local, which requires --stage-id.)
• This recipe serves via --deploy-config. Replica fan-out comes from the deploy config's num_replicas / devices; stage topology comes from the registered WanPipeline.

Example materials¶

#!/usr/bin/env bash
# Serve N independent Wan2.2 TI2V-5B replicas (replica data parallelism).
#
# Usage:
#   NUM_REPLICAS=1 DEVICES=0        ./run_server.sh      # baseline
#   NUM_REPLICAS=2 DEVICES=0,1      ./run_server.sh
#   NUM_REPLICAS=4 DEVICES=0,1,2,3  ./run_server.sh
#
# Then drive load with bench_replica_dp.py (see README.md) and compare
# throughput across replica counts.
set -euo pipefail

MODEL="${MODEL:-Wan-AI/Wan2.2-TI2V-5B-Diffusers}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8098}"
NUM_REPLICAS="${NUM_REPLICAS:-1}"
DEVICES="${DEVICES:-0}"   # one GPU per replica for tensor_parallel_size=1

HERE="$(cd "$(dirname "$0")" && pwd)"
TEMPLATE="$HERE/wan2_2_ti2v_dp.yaml"
CFG="$(mktemp --suffix=_wan2_2_dp.yaml)"
trap 'rm -f "$CFG"' EXIT

sed -e "s/__NUM_REPLICAS__/${NUM_REPLICAS}/" \
    -e "s/__DEVICES__/${DEVICES}/" \
    "$TEMPLATE" > "$CFG"

echo "Serving ${MODEL}: num_replicas=${NUM_REPLICAS} devices=${DEVICES} (config ${CFG})"

vllm serve "$MODEL" --omni \
    --host "$HOST" \
    --port "$PORT" \
    --deploy-config "$CFG"

# Replica data-parallel recipe for Wan2.2 TI2V-5B (video DiT).
#
# Fans out `num_replicas` independent DiffusionEngine replicas, one per GPU,
# and lets the serving layer route each request to a single replica. This
# scales request throughput near-linearly; it does not change single-request
# latency. See ./README.md for measured numbers and the `run_server.sh` driver.
#
# run_server.sh substitutes __NUM_REPLICAS__ / __DEVICES__ from env before serving.
# For tensor_parallel_size = 1, `devices` lists one GPU per replica (pool mode):
# num_replicas=4 -> devices "0,1,2,3".
#
# Stage topology (diffusion DiT, video output) comes from the registered
# WanPipeline, which auto-detects TI2V-5B mode from model_index.json; this file
# carries deployment knobs only.
trust_remote_code: true
distributed_executor_backend: mp

stages:
  - stage_id: 0
    num_replicas: __NUM_REPLICAS__
    devices: "__DEVICES__"
    max_num_seqs: 1
    tensor_parallel_size: 1
    enforce_eager: true
    vae_use_tiling: true
    default_sampling_params:
      seed: 42