vllm.v1.hisparse.layout
¶
Functions:
-
create_hisparse_layout–Size the host pool for groups laid out by
get_hisparse_kv_cache_groups. -
get_hisparse_kv_cache_groups–The groups HiSparse allocates: the host source group, then the GPU
-
get_hisparse_page_layer_configs–The resident and hot layers HiSparse derives from its source layers.
-
get_hisparse_steady_state_concurrency–Max concurrency at max_model_len once running requests read from host.
-
is_hisparse_source_layer–Whether HiSparse backs this layer's KV with host memory: full-attention
_lay_out_hisparse_groups(vllm_config, mla_specs, regular_groups)
¶
Lay out the groups to fill the largest regular page. When a block-outer layout's pool block is not whole indexer pages, either the block grows to whole pages or the pages spread out to divide the block, whichever takes less memory per request. Before the layout is resolved, only the per-layer specs are used.
Source code in vllm/v1/hisparse/layout.py
create_hisparse_layout(vllm_config, groups, host_budget)
¶
Size the host pool for groups laid out by get_hisparse_kv_cache_groups.
Source code in vllm/v1/hisparse/layout.py
get_hisparse_kv_cache_groups(vllm_config, layer_configs)
¶
The groups HiSparse allocates: the host source group, then the GPU
groups sharing one device pool. Derived resident/hot specs in
layer_configs (see resolve_hisparse_specs) are laid out again
from the attention specs they belong to.
Source code in vllm/v1/hisparse/layout.py
get_hisparse_page_layer_configs(vllm_config, mla_specs)
¶
The resident and hot layers HiSparse derives from its source layers. They are per-layer, so they don't depend on the KV cache layout that groups them.
Source code in vllm/v1/hisparse/layout.py
get_hisparse_steady_state_concurrency(vllm_config, kv_cache_config)
¶
Max concurrency at max_model_len once running requests read from host.
A request reading from host pins only its active tail of each resident group, but admitting one still takes its full in-flight window, so the bound is all-but-one requests at steady state plus one being admitted.
Source code in vllm/v1/hisparse/layout.py
is_hisparse_source_layer(layer_config)
¶
Whether HiSparse backs this layer's KV with host memory: full-attention sparse MLA and its indexer.