Skip to content

vllm_omni.config.endpoint_policy

Endpoint restriction policy for omni pipelines.

UNSUPPORTED_ROUTES module-attribute 

UNSUPPORTED_ROUTES: tuple[EndpointRestriction, ...] = (
    EndpointRestriction(
        OmniServingCapability.CHAT_COMPLETIONS_BATCH,
        "Batched chat completions are not yet supported by vLLM Omni.",
    ),
)

EndpointRestriction dataclass

capability instance-attribute 

capability: OmniServingCapability

reason instance-attribute 

reason: str

OmniServingCapability

Bases: Enum

Serving capabilities that pipelines can shut down.

CHAT_COMPLETIONS_BATCH class-attribute instance-attribute 

CHAT_COMPLETIONS_BATCH = RouteTarget(
    "/v1/chat/completions/batch", frozenset({"POST"})
)

COMPLETIONS class-attribute instance-attribute 

COMPLETIONS = RouteTarget(
    "/v1/completions", frozenset({"POST"})
)

methods property 

methods: frozenset[str]

path property 

path: str

RouteTarget

Bases: NamedTuple

A server path & supported methods.

methods instance-attribute 

methods: frozenset[str]

path instance-attribute 

path: str

build_rejection_handler 

build_rejection_handler(reason: str)

Build a rejection handler for a given endpoint for the provided reason.

shutdown_unsupported_routes 

shutdown_unsupported_routes(
    app: FastAPI,
    endpoint_restrictions: tuple[EndpointRestriction, ...],
)

Given an initialized FastAPI server instance and a set of model specific endpoint restrictions, remove the restricted routes and patch a handler that returns 400.