Skip to content

vllm_omni.worker.lmcache_model_runner_mixin ¶

LMCache hidden-state store/restore for the AR stage.

Mixed into GPUARModelRunner only. The shared OmniGPUModelRunner base keeps thin no-op hooks (_setup_lmcache_hidden_state_offload and _drop_hs_pending_state) so non-AR runners carry none of this.

logger `module-attribute` ¶

logger = init_logger(__name__)

LMCacheHiddenStateMixin ¶

Store/restore per-layer hidden states in LMCache alongside KV.

input_batch `instance-attribute` ¶

input_batch: InputBatch

query_start_loc `instance-attribute` ¶

query_start_loc: Any