Skip to content

vllm_omni.config.composable_parallel.routing

Closed routing-pattern hierarchy for declarative parallel strategies.

A routing pattern describes how a request batch is distributed across the workers of one mesh axis (e.g. broadcast to every tensor-parallel rank, hash partition across data-parallel ranks). The hierarchy is closed: subclasses may only be defined in this module, so the set of patterns is a fixed, exhaustively matchable contract.

ROUTING_PATTERN_VARIANTS module-attribute 

ROUTING_PATTERN_VARIANTS: tuple[
    type[RoutingPattern], ...
] = (
    Broadcast,
    PartitionByHash,
    ShardSequence,
    ShardSpatial,
    RouteByExpert,
    PipelineMicrobatch,
    DuplicateWithCondTag,
    RouteByStage,
)

RouterFn module-attribute 

RouterFn = Callable[..., object]

Broadcast dataclass

Bases: RoutingPattern

Every worker in the group sees the whole batch (TP, dense EP).

pattern_kind 

pattern_kind() -> str

DuplicateWithCondTag dataclass

Bases: RoutingPattern

Duplicate batch with a conditional tag (CFG).

flag_name class-attribute instance-attribute 

flag_name: str = 'guidance_branch'

pattern_kind 

pattern_kind() -> str

PartitionByHash dataclass

Bases: RoutingPattern

Stable hash partition by request key (DP, HSDP).

key class-attribute instance-attribute 

key: str = 'request_id'

pattern_kind 

pattern_kind() -> str

PipelineMicrobatch dataclass

Bases: RoutingPattern

Pipeline parallelism microbatch routing (PP).

microbatch_size class-attribute instance-attribute 

microbatch_size: int = 1

pattern_kind 

pattern_kind() -> str

RouteByExpert dataclass

Bases: RoutingPattern

Sparse MoE expert routing.

router_id class-attribute instance-attribute 

router_id: str = 'default'

pattern_kind 

pattern_kind() -> str

RouteByStage dataclass

Bases: RoutingPattern

Route requests by pipeline stage policy (stage replicas).

The random/round_robin/least_queue policies describe stateless load balancing that a runtime (e.g. an engine's built-in DP load balancer, or the omni StagePool balancer) can realize. hash describes deterministic, key-stable routing that a load balancer cannot honor on its own (see PartitionByHash).

routing_policy class-attribute instance-attribute 

routing_policy: Literal[
    "random", "round_robin", "least_queue", "hash"
] = "round_robin"

pattern_kind 

pattern_kind() -> str

RoutingKeyError

Bases: KeyError

Raised when a routing key is missing from a request.

missing_key instance-attribute 

missing_key = missing_key

request_id instance-attribute 

request_id = request_id

RoutingPattern dataclass

Base class for closed routing patterns. Subclass only within this module.

pattern_kind 

pattern_kind() -> str

ShardSequence dataclass

Bases: RoutingPattern

Shard along a sequence dimension (SP-Ulysses, SP-Ring, CP).

dim class-attribute instance-attribute 

dim: int = 1

pattern_kind 

pattern_kind() -> str

ShardSpatial dataclass

Bases: RoutingPattern

Shard along a spatial grid (VAE-PP).

grid class-attribute instance-attribute 

grid: tuple[int, int] = (1, 1)

pattern_kind 

pattern_kind() -> str