Base Model Interfaces

Module Contents

class vllm.model_executor.models.interfaces_base.VllmModel(vllm_config: VllmConfig, prefix: str = '')

The interface required for all models in vLLM.

class vllm.model_executor.models.interfaces_base.VllmModelForTextGeneration(vllm_config: VllmConfig, prefix: str = '')

The interface required for all generative models in vLLM.

compute_logits(hidden_states: T, sampling_metadata: SamplingMetadata) → T | None

Return None if TP rank > 0.

sample(logits: T, sampling_metadata: SamplingMetadata) → SamplerOutput

Only called on TP rank 0.

class vllm.model_executor.models.interfaces_base.VllmModelForPooling(vllm_config: VllmConfig, prefix: str = '')

The interface required for all pooling models in vLLM.

pooler(hidden_states: T, pooling_metadata: PoolingMetadata) → PoolerOutput

Only called on TP rank 0.