Pipeline and deploy configurations¶

In vLLM-Omni, a model's PipelineConfig defines its fixed stage topology, while a deploy configuration controls how those stages run.

For diffusion sampling, explicit request fields override --default-sampling-params, which overrides deploy YAML defaults. CLI values are merged by stage ID, and extra_args is merged by key. Invalid CLI JSON fails startup; model-owned sampling constraints still apply.

Pipeline configuration¶

PipelineConfig and its StagePipelineConfig entries are Python definitions owned by the model implementation and registered in OMNI_PIPELINES. They describe fixed model topology and are not accepted in deploy YAMLs.

Common PipelineConfig fields include:

Common StagePipelineConfig fields include:

To add or change topology, define and register a new pipeline variant. Use deploy YAML only for runtime placement, resource sizing, connectors, and other deployment overrides.

Engine output types¶

engine_output_type is parsed when the LLM stage's output processor is built during service initialization. Invalid values fail startup before requests are accepted; engine replicas may already have been initialized at that point and are cleaned up on failure.

Use the exact lowercase names listed above. Combinations such as text+audio or text,audio remain supported, with every component canonical. Omitted or None values retain the default behavior (TEXT in the output processor). Empty strings, whitespace, case variants, and aliases are rejected.

For existing custom pipelines or direct callers, replace old aliases explicitly:

token_ids is a separate type, not a text alias. It describes discrete vocabulary or codebook indices, distinct from continuous latent representations. Existing models using latent retain their historical payload contracts. GLM-Image's AR stage keeps engine_output_type="token_ids"; generated IDs still travel through cumulative_token_ids, and source-image IDs through ids.prior_image.

Bare primary tensors and generic producer keys (model_outputs or hidden) use the declared type as their default output key, including token_ids. The normalizer does not inspect or convert their dtype. Distinct auxiliary outputs must use explicit field names: model_outputs and hidden both map to the same default key and must not represent two independent outputs. GLM-Image disables auxiliary hidden output for its default GPU execution. Some runner paths still attach it; that field is then named token_ids even though it contains hidden states. The GLM-Image bridge ignores this auxiliary field and reads generated IDs from cumulative_token_ids instead.

Token-ID tensors use the existing dimension-0 accumulation default, subject to registered per-key overrides, and remain available after DELTA emissions. This is separate from the existing accumulation of cumulative_token_ids. final_output_type and Diffusion's output_type are separate settings and are unchanged by these rules.

Deploy configuration schema¶

The new deploy schema lives under vllm_omni/deploy/ and is paired with a frozen PipelineConfig registered by the model's pipeline.py. Each deploy YAML has these top-level fields:

Duplex replica placement¶

The following fields belong under duplex_session and apply when session_mode: duplex:

For example, a two-stage local pipeline with two replicas per stage can use:

session_mode: duplex
duplex_session:
  max_sessions: 4
  replica_policy: least_sessions
  max_sessions_per_replica: 2
stages:
  - stage_id: 0
    num_replicas: 2
    devices: "0,1"
  - stage_id: 1
    num_replicas: 2
    devices: "2,3"

This example assumes one GPU per replica. With equal replica counts and no explicit groups, group g maps every stage to replica g. Unequal counts require explicit groups. Every group must cover every pipeline stage, all replica IDs must exist, every replica must belong to at least one group, and duplicate groups are rejected. A replica shared by multiple groups has one shared capacity counter; the global limit can therefore exceed the capacity that the groups can actually admit.

Placement selects an eligible group by its highest reserved-to-capacity ratio, then its total reserved count, with round-robin tie-breaking. Sessions keep their group across turns, cancellation, epochs and reconnects. Idle sessions hold their slots, and closing sessions release them after cleanup completes. When no group has capacity, opening a session fails with the retryable resource_exhausted error containing replica_capacity_exhausted.

least_sessions supports local single-host deployments only; coordinator or distributed routing is rejected at startup. A per-replica cap above 1 also requires the model plugin to support concurrent sessions on one replica; use 1 for plugins without that capability, including VoiceChat. A failed replica closes all sessions assigned to it, including idle sessions. Those sessions are not migrated, and other groups continue serving. Under least_sessions, workers receive max_sessions_per_replica as duplex_max_sessions; under legacy, they continue to receive the global max_sessions value.

Multiple duplex frontends¶

Use --api-server-count K to run separate API and duplex orchestrator processes on one listening port. The default remains one frontend. Multiple frontends require replica_policy: least_sessions and a plugin whose capabilities declare supports_session_resume: false, such as PersonaPlex. Resume-capable models and the legacy policy remain unsupported with multiple duplex frontends.

Group g (its zero-based position in replica_groups, or its implicit replica index) belongs to frontend g % K. K cannot exceed the number of groups. Explicit groups sharing any stage replica must have the same frontend owner; cross-owner sharing is rejected. Each frontend places sessions only in its owned groups, while workers still use max_sessions_per_replica as their cap.

The global max_sessions budget is split evenly: frontend i receives max_sessions // K + (i < max_sessions % K) slots, bounded further by its owned replica capacity. This bound is the smallest owned replica count across stages multiplied by max_sessions_per_replica. Unused slots are not borrowed by another frontend. Choose max_sessions >= K; otherwise frontends assigned zero slots accept no sessions and skip warmup. For a single frontend, the existing global and per-replica admission checks are unchanged.

For example, copy vllm_omni/deploy/personaplex.yaml, set max_sessions: 64, replica_policy: least_sessions and max_sessions_per_replica: 8 under duplex_session, and set num_replicas: 8 and devices: "0,1,2,3,4,5,6,7" on both existing stages. With one GPU per replica this puts a complete two-stage group on each GPU. Pass --api-server-count 4 to give each frontend two groups and a session budget of 16. Model memory settings still need to fit the target GPU; frontend sharding does not change them.

Connections are distributed by the operating system, not by available session capacity. A frontend can reject a new session while another has free slots; capacity errors include retryable: true, so clients should back off and open a new connection. /health returning 200 does not guarantee admission capacity on the frontend accepting a connection. PersonaPlex startup warmup remains opt-in: set duplex_session.warmup_frames to a positive frame count to warm each frontend's owned groups locally. See serving guidance for readiness and failure behavior.

For fields whose deploy default is null, the deploy layer contributes no override. The effective value may still come from a platform section, an explicit CLI or stage override, or the downstream vLLM engine default.

Note: for the diffusion path, an omitted distributed_executor_backend selects uni on a single GPU (in-process worker, no MessageQueue / /dev/shm output segments) and mp when num_gpus > 1. Set mp explicitly to keep a worker subprocess on one GPU. Set ray for Ray-managed single- or multi-node diffusion workers. external_launcher is not supported yet.

The diffusion ray backend runs one worker actor per GPU and supports workers spanning Ray nodes. Enable it in the deploy YAML:

distributed_executor_backend: ray

The executor reuses Ray when already initialized; otherwise it calls ray.init(), which handles cluster discovery or local startup and respects RAY_ADDRESS. There is no diffusion-specific address setting. Ray must be installed on every node.

Ray diffusion workers receive the driver's VLLM_*, OMNI_*, DIFFUSION_*, NCCL_*, TORCH_NCCL_*, UCX_*, and Hugging Face environment settings, plus PYTHONPATH, CUDA_LAUNCH_BLOCKING, and OMP_NUM_THREADS. Explicit stage runtime.env entries are also forwarded and take precedence. Device visibility, rank/rendezvous settings, worker host addresses, and RAY_* variables are excluded so each worker retains its own assignment and identity.

For multiple nodes, start the cluster before launching Omni. For example, with head IP 10.0.0.1:

# On the head node:
ray start --head --node-ip-address=10.0.0.1 --port=6379

# On each additional worker node:
ray start --address=10.0.0.1:6379

# On the host launching Omni (MODEL and my_deploy.yaml are placeholders):
RAY_ADDRESS=10.0.0.1:6379 vllm serve MODEL --omni --deploy-config my_deploy.yaml

Omni creates actors on the resources of that cluster; it does not start Ray on additional machines.

Stage CPU placement¶

On Linux, stages can use vLLM's NUMA settings to select CPU lists. Add the settings to the existing stage entries in your deploy configuration:

stages:
  - stage_id: 0
    numa_bind: true
    numa_bind_nodes: [0]
    numa_bind_cpus: ["2-3"]
  - stage_id: 1
    numa_bind: true
    numa_bind_nodes: [0]
    numa_bind_cpus: ["4-7"]

These CPU and NUMA indices are examples; choose indices available to the process or container. Use one CPU-list entry per visible GPU. Binding requires numactl and VLLM_WORKER_MULTIPROC_METHOD=spawn.

With an in-process executor (uni), the worker's CPU list applies to the stage EngineCore itself. With a multiprocess executor (mp), workers use their CPU lists and EngineCore retains its wider NUMA-node binding. Choose CPU sets away from competing workloads; affinity does not reserve CPUs exclusively.

Stage-level runner selection¶

model_runner: v1 or v2 at the deploy level sets the default runner. A model_runner on an individual stage overrides that default, allowing one stage to migrate or roll back independently. This does not change PipelineConfig topology, input adapters, or the connector payload contract. An omitted stage override preserves the deploy-level selection.

MRv2 native downstream receivers accept turn-based requests. A pipeline that sets supports_duplex_mrv2 also accepts duplex sessions on those stages. Qwen3-Omni and MiniCPM-o 4.5 set that flag. Resumable streaming input uses the V1 prompt-replacement path unless the pipeline implements the MRv2 equivalent. Selecting a runner does not add session capabilities the pipeline lacks. Platform fallbacks and stage overrides are validated after configuration resolution.

Stage fields¶

Operator backends¶

AR stages accept attention_backend, moe_backend, and linear_backend. Diffusion stages share moe_backend and linear_backend, but use their own diffusion_attention_config (or the deprecated diffusion_attention_backend shorthand), not AR's attention_backend.

These backend fields are stage-only: set them under each stages entry, not at the deploy YAML top level, where they are ignored.

For a pipeline with AR stage 0 and diffusion stage 1, the deploy fragment is:

stages:
  - stage_id: 0
    attention_backend: TRITON_ATTN
    moe_backend: auto
    linear_backend: torch
  - stage_id: 1
    diffusion_attention_config:
      default: TORCH_SDPA
    moe_backend: auto
    linear_backend: torch

The deploy fields default to null (no override). Effective MoE/linear defaults remain auto, and explicit auto remains distinct from omission. KernelConfig supplies validation and case/hyphen normalization. AR attention continues to use upstream enum parsing and auto/None semantics.

CLI/runtime values retain their existing precedence. A first-class backend field wins over the same engine_extras key, with a warning on conflict; extras-only configurations remain supported. Existing backend fallback rules are unchanged, and this PR does not claim support for unvalidated platforms. For typed diffusion stages, an explicitly supplied model-config backend wins over a different explicitly supplied diffusion-config backend, with a warning. In vLLM 0.29, unquantized ROCm GEMM dispatch does not consult linear_backend; NPU plugin behavior is not validated here. Plain torch.nn.Linear layers do not consume vLLM's kernel configuration.

An offline CUDA component smoke test runs a tiny, randomly initialized native FLUX transformer, checks the actual linear callable and SDPA implementation, and executes a forward pass (no weights download or image-quality claim):

pytest -s -v tests/diffusion/models/flux/test_backend_selection.py -m 'local_model and cuda'

The CPU config regression suite is collected by the existing L1 job:

pytest -s -v tests/config/test_backend_selection.py -m 'core_model and cpu'

Both commands require matching vLLM/vLLM-Omni dependencies. The component smoke test additionally needs one CUDA GPU; it is opt-in and not a performance claim.

Field reference¶

Each entry under stages: accepts any StageDeployConfig field directly (no nested engine_args:). Only fields whose value legitimately varies across stages live here; pipeline-wide settings (trust_remote_code, distributed_executor_backend, dtype, quantization, prefix/chunked prefill, DP/PP sizes) are declared at the top level and applied to every stage. Unknown keys fall through to engine_extras: and are forwarded to the engine. Frequently used fields are listed below; the source-of-truth schema is StageDeployConfig in vllm_omni/config/stage_config.py.

Note: a stage's devices count must equal its local world size (tensor_parallel_size × data_parallel_size_local × pipeline_parallel_size, falling back to global data_parallel_size when the local size is unset), or num_replicas × that product for a replica pool. A mismatch fails early and names the offending stage. A top-level --tensor-parallel-size is broadcast to every stage, so it can make a single-GPU stage violate this contract ( issue #5003); fix that case with --stage-overrides (set tensor_parallel_size and devices together per stage) or set TP only on the multi-GPU stage.

Connector schema¶

Each entry under top-level connectors: follows this shape:

connectors:
  <connector_name>:
    name: <ConnectorClassName>     # required — class registered in vllm_omni.distributed
    extra:                         # optional — forwarded to the connector's __init__
      <key>: <value>
      # Additional connector-specific options

A stage references a connector by name in its input_connectors / output_connectors:

connectors:
  shm:
    name: SharedMemoryConnector

stages:
  - stage_id: 0
    output_connectors: {to_stage_1: shm}
  - stage_id: 1
    input_connectors:  {from_stage_0: shm}

CLI flags¶

Stage-Based CLI Paradigm¶

The stage-based CLI paradigm facilitates the execution of discrete pipeline stages within isolated processes:

• Stage 0 typically encapsulates the orchestrator and the primary API server. Invocation requires --stage-id 0, --omni-master-address, --omni-master-port, and standard port declarations (e.g., --port).
• Worker Stages operate without a distinct API server (i.e., using --headless), are assigned sequential --stage-id identifiers, and must reference the corresponding --omni-master-address and --omni-master-port parameters to successfully register with Stage 0.

You may also explicitly pass --stage-id none to run only the API server and orchestrator if you want to run every stage as headless.

For migrated architectures, the system automatically resolves and loads the bundled deployment YAML. Consequently, the primary execution path does not necessitate the explicit definition of --deploy-config: the example below uses CUDA_VISIBLE_DEVICES=0 for Stage 0 and CUDA_VISIBLE_DEVICES=1 for Stage 1.

CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --port 8091 \
    --stage-id 0 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 1 \
    --headless \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

When instantiating a custom deployment YAML, append the --deploy-config /path/to/override.yaml directive to all node invocations.

In the context of standard initialization architectures, utilizing the --stage-overrides parameter operates as the optimal methodology for delineating stage-specific tuning from the CLI interface:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --stage-overrides '{"1": {"gpu_memory_utilization": 0.5}}'

Conversely, in the context of the stage-based CLI paradigm, given that each execution process exclusively instantiates a single pipeline stage, configuration override attributes can be defined uniformly via explicit CLI flags on the corresponding instantiation command, rendering composite --stage-overrides JSON strings unnecessary:

CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 1 \
    --headless \
    --gpu-memory-utilization 0.5 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

Precedence¶

From highest to lowest:

1. Per-stage overrides (--stage-overrides JSON)
2. Explicit global CLI flags (--gpu-memory-utilization 0.85, etc.)
3. Platform section (platforms.npu.stages, etc.) on top of the base stages:
4. Overlay YAML (via base_config:) on top of the base YAML
5. Parser defaults

vLLM LoRA on LLM stages¶

AR and generation stage entries accept vLLM's LoRA engine options, such as enable_lora, max_lora_rank, max_loras, max_cpu_loras, lora_target_modules, and fully_sharded_loras. As in vLLM, the other options take effect only on a stage that also sets enable_lora: true.

stages:
  - stage_id: 0
    enable_lora: true
    max_lora_rank: 64

The equivalent CLI and Python forms are --stage-overrides '{"0": {"enable_lora": true, "max_lora_rank": 64}}' and Omni(model, stage_overrides={"0": {"enable_lora": True, "max_lora_rank": 64}}).

• Top-level --enable-lora (or enable_lora=True) works only on single-stage pipelines. Multi-stage pipelines reject global vLLM LoRA options and must use stage-scoped keys, including in the stage-based / --headless CLI, where each process resolves the whole pipeline.
• AsyncOmni.generate(..., lora_request=...) applies the adapter on stage 0.
• Diffusion stages reject enable_lora and configure LoRA through lora_path, lora_scale, and lora_backend instead; see Diffusion LoRA.
• max_cpu_loras sizes the CPU LoRA cache of whichever stage type receives it. A global value on a pipeline with a diffusion stage sizes only the diffusion cache; set an LLM stage's value with a stage-scoped key.

The design is described in LoRA.

Worked override example¶

Starting from the bundled vllm_omni/deploy/qwen3_omni_moe.yaml:

# vllm_omni/deploy/qwen3_omni_moe.yaml (excerpt)
async_chunk: true
stages:
  - stage_id: 0
    gpu_memory_utilization: 0.9
    max_num_seqs: 32
  - stage_id: 1
    gpu_memory_utilization: 0.7
    max_num_seqs: 16

A user-authored overlay that inherits the base and overrides only stage 1:

# my_overrides.yaml
base_config: /path/to/vllm_omni/deploy/qwen3_omni_moe.yaml
stages:
  - stage_id: 1
    gpu_memory_utilization: 0.5     # smaller GPU

Launched with both an explicit global flag and a per-stage override:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --deploy-config my_overrides.yaml \
    --max-model-len 16384 \
    --stage-overrides '{"0": {"max_num_seqs": 8}}'

Within the stage-based CLI paradigm, equivalent configuration parameters can inherently be passed directly as command-line arguments to the designated single-stage process instantiation:

CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 0 \
    --max-num-seqs 8 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

Effective config per stage after the merge:

Therefore, as a core part of vLLM-Omni, a model's pipeline and deployment configurations have several main functions:

• Claim partition of stages and their corresponding class implementation in model_executor/models.
• The disaggregated configuration for each stage and the communication topology among them.
• Engine arguments for each engine within the stage.
• Input and output dependencies for each stage.
• Default input parameters.

To override specific parameters, explicitly inject the customized configuration schema in both online and offline instantiation flows. Use the --deploy-config flag when loading a deploy configuration.

Examples:

For offline inference (assuming the necessary dependencies have been imported):

model_name = "Qwen/Qwen2.5-Omni-7B"
omni = Omni(model=model_name, deploy_config="/path/to/deploy_config.yaml")

For online serving:

vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml

Qwen3-TTS with Model Runner V2¶

Qwen3-TTS runs the native CUDA Model Runner V2 pipeline by default on vLLM 0.29.0. MRV2 is an experimental feature for this model: the bundled default profile selects it on CUDA only, and its scheduler and delivery paths are still being qualified. Set model_runner: v1 in a copy of the deploy config to opt out; the platforms: sections of qwen3_tts.yaml keep V1 on NPU, XPU, ROCm and MUSA. Select one of these deployment profiles:

# MRV2 is the default; pass a copy with `model_runner: v1` to force V1.
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni \
  --deploy-config /path/to/qwen3_tts_v1.yaml

The V2 profiles bound Talker prefill to 512 tokens per step and select the Talker AR runner and the Code2Wav generation runner together. Native inter-stage delivery carries codec payloads directly; request-owned snapshots preserve buffers through asynchronous completion and CUDA graph reuse. Terminal completion waits for upstream stage metrics before releasing request state. Platform sections retain V1 on NPU, XPU, ROCm and MUSA; this change does not qualify MRV2 on those backends or enable other model families. Selecting V2 for a stage without supports_native_mrv2_data_plane emits a warning when pipeline and deployment settings are merged. That stage retains the legacy transport path; the warning does not establish support for that combination.

MRV2 model hooks use capability declarations rather than architecture or stage names. Omni lifecycle flags select the model state, and _returns_tuple declares the capture output contract. The optional batched predictor hook is mtp, with an explicit mtp_output_key (a string or two-part payload key), optional mtp_validity_key, and mtp_graph_safe/mtp_disable_graph capture controls. Its inputs are token IDs, embeddings, previous hidden states and per-row conditioning; it returns updated embeddings and prediction codes. Models may supply mtp_sampling_params and get_mtp_seed(sampling_params) for model-local explicit seeds, and declare mtp_accepts_per_row_generators, mtp_accepts_req_infos, or mtp_sample_uniforms (with mtp_sample_steps and mtp_sample_vocab_size) as needed. Qwen3-TTS retains its existing talker_mtp entry point for V1.

Selecting one or two Qwen3-TTS stages¶

Choose the deployment profile explicitly with --deploy-config. The default qwen3_tts pipeline retains two stages: the Talker publishes codec chunks to Code2Wav, which produces audio. Its stage devices can be placed on the same GPU or on separate GPUs. qwen3_tts_fused is a separate, opt-in pipeline for one CUDA GPU: the Talker owns the stateful codec decoder and emits PCM directly, without a Code2Wav engine or stage connector.

# Two stages on one GPU; the two-stage pipeline remains independently selectable.
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
  --deploy-config vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2_single_gpu.yaml

# One stage on one GPU; no inter-stage transport or Code2Wav scheduling.
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
  --deploy-config vllm_omni/deploy/qwen3_tts_fused_single_gpu.yaml

The single-stage profile requires CUDA, V2, asynchronous chunks, TP=PP=1, an in-process worker and disabled prefix caching. Select the full profile; adding talker_stream_decode: true to a two-stage Talker is rejected because its latent output belongs to the Code2Wav input contract. Single-stage options live in stages[].additional_config; two-stage transport and first-frame options live in connectors.*.extra.

The single-stage profile defaults talker_stream_first_audio to false for throughput. To opt into a separate first-frame delivery queue, save this overlay beside the bundled deployment files and pass its path to --deploy-config:

base_config: qwen3_tts_fused_single_gpu.yaml
stages:
  - stage_id: 0
    additional_config:
      talker_stream_decode: true
      talker_stream_first_audio: true
      ref_code_context_frames: 72
      code_predictor_kv_cache: true
      code_predictor_fused_sampling: true
      code_predictor_fused: true

This queue uses PCM from the single-stage decoder. The two-stage talker_first_audio option below uses its own first-frame decoder and does not require single-stage execution. The two options cannot be enabled together. Single-stage prefix-cache reset with reset_running_requests=True returns False while requests are active; wait for completion or abort them before retrying.

Reference-cloning context is selected independently by each profile. The single-stage profile explicitly uses 72 reference frames, matching the qwen3_tts.yaml context boundary. A shorter additional_config.ref_code_context_frames (for example 25) can change timbre continuity even when WER is similar; the throughput measurements for CustomVoice do not qualify Base voice cloning quality. Reference layout, grouping and priming are model hooks; the shared worker does not select Qwen's context length or codebook layout.

Qwen3-TTS first-frame delivery and rollback¶

The standard qwen3_tts.yaml, qwen3_tts_high_concurrency.yaml, qwen3_tts_mrv2.yaml and qwen3_tts_high_concurrency_mrv2.yaml profiles enable the talker_first_audio connector option, as does the experimental two-stage qwen3_tts_high_concurrency_mrv2_single_gpu.yaml profile. This changes the default CUDA MRV2 streaming path: residual prediction runs eagerly after the Talker sample, the Talker loads an additional first-frame decoder with its weights and CUDA graphs, and the orchestrator orders its audio before subsequent Code2Wav chunks.

Code2Wav retains full-audio prefix graphs alongside the optional state-only graphs. Each request's delivery marker selects the graph and audio trimming; enabling the option alone never suppresses audio or disables prefix batching for requests that retain regular codec delivery.

The path requires asynchronous chunks, TP/PP 1, an in-process executor and disabled prefix caching. Requests with reference codes and unsupported runners or platforms retain regular Code2Wav delivery.

To restore regular codec delivery and avoid loading the Talker's additional decoder, use a deploy overlay with the relevant base profile:

base_config: qwen3_tts.yaml
connectors:
  connector_of_shared_memory:
    extra:
      talker_first_audio: false

First-packet latency measures when PCM starts arriving. Time to first audible audio (TTFA) also includes any leading silence in the generated audio. An earlier first packet therefore does not necessarily improve TTFA; measure both for the intended voice and workload.

Included performance work¶

• Request snapshots have a fast path for immutable scalar leaves, including waveform lists. Mutable containers, aliases and cycles keep deep-copy semantics. This is not the historical shallow dict(prompt) experiment.
• The API reference cache holds owned float32 arrays with a default capacity of 1024 entries and a 512 MiB waveform-payload budget. Cache hits return independent lists. Qwen3-TTS model artifact caches also default to 1024 entries. These caches have different owners and eviction policies; equal entry limits do not make them a single coherent cache.
• Talker state stays on GPU where its lifecycle allows it. BOS/EOS projections are cached in their projection dtype and invalidated on weight loading.
• Code2Wav packs CPU codec inputs before device transfer. Optional B2/B4 graph buckets batch compatible requests without sharing their per-request state.
• Native output materialization runs in a bounded worker and drains before closing the data plane. The existing control-only shortcut avoids launching model work for a step that has only lifecycle events.

MTP prefix re-prefill remains disabled in the high-concurrency V2 profile. Adding more graph shapes increases compilation cost and has not established a stable end-to-end gain for this extracted version.

Decoder batches and playback¶

Changing decode_cudagraph_batch_sizes selects the captured batch buckets. Keep decode_batch_max_size consistent with the intended maximum too; the stateful graph path currently groups according to the captured buckets, while the stateless path also uses the explicit maximum.

B4 remains experimental. First-packet latency alone does not establish uninterrupted playback. Validate inter-chunk arrival times, buffering, WER and speaker similarity before adopting either batching preset for a production workload. Floating-point decoder outputs can differ across batch sizes; this PR does not claim bitwise or quality equivalence.

Experimental MPS deployment¶

NVIDIA MPS lets colocated CUDA stage processes share GPU execution resources. It is disabled by default. The experimental qwen3_tts_high_concurrency_mrv2_single_gpu.yaml profile sets cuda_mps: true, alongside cached residual prediction, fused sampling, time-major codec convolutions, first-frame delivery, and larger graph batches.

A missing nvidia-cuda-mps-control binary, an MPS daemon that fails to start or answer get_server_list (including an unreachable operator-managed CUDA_MPS_PIPE_DIRECTORY), a stage that is not one numeric CUDA GPU, parallel_stage_init, or a diffusion/remote stage disables MPS for that configuration and continues. Use numeric GPU ordinals for stage placement and visibility so initialization locks identify the physical GPU before MPS remaps it. When MPS does start, it uses a private daemon for each selected GPU and stops its own daemon after the stages exit. If CUDA_MPS_PIPE_DIRECTORY already names an operator-managed daemon, the runtime reuses it without stopping it. Diffusion and remote stages are unsupported. Set cuda_mps: false in a deploy overlay to disable automatic MPS management.

Stage env.CUDA_MPS_PIPE_DIRECTORY overrides the parent setting when selecting the daemon; an explicit empty string selects a private daemon. Stages sharing a GPU reuse the first stage's daemon. Later stages may omit the setting or explicitly match that policy; a conflicting explicit setting fails before that stage starts. The parent environment is unchanged.

MPS can improve throughput under concurrent load while increasing first-packet or first-audible-audio latency, particularly at low request rates. For a latency-sensitive workload, compare the same profile with cuda_mps: false. An inherited operator-managed MPS daemon must also be disabled by its owner for that comparison; this switch only controls automatic MPS management.

MPS does not reserve a GPU. Use only assigned GPUs, explicitly place stages on the intended GPU, and warm the complete pipeline before measuring performance.

MOSS-TTS Local 1.5 with Model Runner V2¶

OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 uses one default deployment, moss_tts_local.yaml. Omitting --deploy-config selects the same file:

vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --deploy-config vllm_omni/deploy/moss_tts_local.yaml

On CUDA, both stages use MRV2 and 128 stream slots on one GPU, with full-quota private MPS, a 32 GiB Talker KV budget, prefix caching and Triton attention. The Talker bounds prefill to 512 tokens and uses FULL CUDA graphs. The codec uses triton_slot, Inductor compilation, codec-owned graphs through batch 128 and dispatch coalescing at 16 rows or 6 ms. Initial and steady chunks remain 1 and 15 codec frames; the Stage0 first-frame path follows the Talker capture sizes and safe-request/executor requirements.

These CUDA settings are validated on H200 and require memory for both stages and their graph/state pools, plus nvidia-cuda-mps-control on PATH. Smaller GPUs and services without MPS should provide deployment overrides for the capacities, capture sizes, KV budget and platforms.cuda.cuda_mps. The pipeline no longer changes configuration based on a GPU-memory or MPS probe.

Other platforms use V1 in the same file; NPU keeps its codec graph override. On Ascend NPU, moss_tts_local_npu_mrv2.yaml is an opt-in profile that runs the Talker (Stage0) and codec (Stage1) on MRV2. The Talker builds Ascend attention metadata, and the codec inherits the NPU streaming graph settings from moss_tts_local.yaml. The Stage0 first-frame direct decode is off on NPU because it needs CUDA-only kernels, so Stage1 delivers every chunk:

vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --deploy-config vllm_omni/deploy/moss_tts_local_npu_mrv2.yaml

The 2/3-NPU files retain their distinct multi-device topology. These changes do not enable MRV2 for MOSS Delay, Realtime or Nano. Local 1.5 emits 48 kHz stereo; benchmark raw PCM with VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=48000 and VLLM_OMNI_BENCH_AUDIO_CHANNELS=2. See the MOSS recipe for the execution path and configuration overrides.