Configuration

Environment Variables

Variable	Default	Description
VLLM_METAL_MEMORY_FRACTION	auto	Metal memory budget mode; see Paged KV vs MLX KV Memory Settings
VLLM_MLX_DEVICE	gpu	MLX device (gpu or cpu)
VLLM_METAL_USE_PAGED_ATTENTION	1	Enable experimental paged KV cache
VLLM_METAL_DISABLE_NAX	0	Emergency override for automatic M5 NAX prefill attention. Set to 1 to force the non-NAX fallback.
VLLM_METAL_MULTIMODAL_MODE	auto	Multimodal serve mode: auto uses the compatibility allowlist; multimodal-native disables overrides
VLLM_USE_MODELSCOPE	False	Set True to change model registry to https://www.modelscope.cn/
VLLM_METAL_MODELSCOPE_CACHE	None	Specify the absolute path of the local model
VLLM_METAL_GDN_LAZY_KERNELS	1	Enable lazy GDN kernels for eligible hybrid batches. Set to 0 to force the eager conv / C++ recurrent fallback path.
VLLM_METAL_DECODE_PIPELINE	1	One-step-ahead decode sampling pipeline: eligible pure-decode greedy steps defer the sampling sync one step so the next step's graph build and submit overlap the in-flight GPU forward. Greedy output is unchanged. Disabled automatically when speculative decoding is configured. Set to 0 to force the fully synchronous per-step sample path.
VLLM_METAL_COMPILED_MLP	0	Opt-in compiled stateless-MLP dispatch: decode-shaped MLP/MoE block calls run through an mx.compile trace, fusing the per-layer elementwise glue and cutting the per-step op count. Outputs are bitwise identical to the eager dispatch for quantized checkpoints (unquantized fp16 fusion may differ at the ulp level). LoRA serves keep the eager path. Off by default while the dispatch gathers serve mileage; set to 1 to enable.
VLLM_METAL_NATIVE_SAMPLING	0	Opt-in MLX-native non-greedy sampling for the decode pipeline: the pipeline's deferred sampler learns a temperature/top-k/top-p graph (mx.random.categorical over a top-k/top-p mask), so eligible non-greedy pure-decode steps defer their sampling sync like greedy ones instead of dropping to the synchronous torch path. Requests with seeds, penalties, logprobs, or token constraints keep the torch path (min_p and logit_bias are rejected at request validation on Metal, as before). Sampled tokens come from the MLX RNG keyed off the engine seed, so outputs differ from the torch sampler's stream. Off by default while the path gathers serve mileage; set to 1 to enable.
VLLM_METAL_MLA_KERNEL	0	Enable the experimental absorbed-MLA single-pass Metal decode kernel (RFC #360). Off by default; the MLA wrapper falls back to the MLX SDPA per-request slow path. Set to 1 to route absorbed-MLA decode through the kernel when the workload matches the instantiated specialization (kv_lora_rank=512, qk_rope_head_dim=64, block_size ∈ {16, 32}, fp16/bf16, decode-only).
VLLM_METAL_BUILD_FROM_SOURCE	0	Compile the native _paged_ops Metal extension from source at runtime instead of loading the prebuilt artifact shipped in the wheel. For kernel developers / source installs; requires the Xcode command-line tools (clang++). Off by default — release wheels ship the .so prebuilt. See Contributing.
VLLM_METAL_SPEC_VERIFY_WINDOW	0	Enable spec-decode verification window mode (issue #465): the K+1 verification rows share each KV block load instead of re-reading the context per row. Off by default; verify windows keep the expanded per-token layout. Outputs are bitwise identical either way; the win is chip- and shape-dependent (measured up to +40% e2e at concurrency 16-32 with 8k contexts on M2/M3 Ultra, and regressions single-stream on M4 Pro and at concurrency 32 on M2 Max). MLA, hybrid-GDN, and head sizes above 256 always use the expanded layout. The same opt-in also merges the draft_model proposer's small committed-token ingest into one window per request (head sizes above 256 keep the expanded layout there too); single-stream TPOT is within run-to-run noise of the expanded ingest (0.6B pair, 8k prefix, M4 Pro), and generated tokens are identical.
VLLM_METAL_VISIBLE_DEVICES	—	Set automatically by the Ray executor per worker (the device-control var); not user-configurable. See Distributed.
VLLM_METAL_RING_BASE_PORT	32323	Base TCP port for the MLX ring data plane under pipeline parallelism; stage r binds base + r (so the default is 32323/32324 for two stages). Set the same value on every node to move the ring off a busy port — e.g. when an mlx.launch job, a restart still in TIME_WAIT, or another PP job holds the default. See Distributed.

MLX Command-Buffer Defaults

On macOS the plugin defaults MLX_MAX_OPS_PER_BUFFER to 2000 via setdefault, so a value you export yourself always wins. MLX's own default is sized for small generate loops; a vLLM decode step on a large MoE model builds thousands of lazy ops per step, and the resulting per-buffer commit overhead slows the step submit. 2000 sits on the measured plateau.

MLX_MAX_MB_PER_BUFFER trades transient profile memory for per-step commit overhead. The plugin defaults it to 2000 when the usable budget (total memory times the effective memory fraction) is at least 90 GiB, and leaves it unset below that, on Ray executors, or when max_num_batched_tokens exceeds 4096 (the #585 startup-failure shape). A value you export yourself always wins. Outputs are unaffected.

Multimodal Serve Modes

• auto: use the text-only compatibility path for checkpoints on the compatibility allowlist, such as Gemma4 and Qwen3.5/Qwen3.6 FP8 conditional-generation wrappers.
• multimodal-native: disable the compatibility fallback and keep the native multimodal path active when validating or developing real multimodal support.

Speculative Decoding

Pass --speculative-config with a JSON object to enable speculative decoding. Use --no-async-scheduling (required for all spec-decode methods on Metal). See Speculative Decoding for supported methods, model pairing, and memory considerations.

Paged KV vs MLX KV Memory Settings

• MLX path (VLLM_METAL_USE_PAGED_ATTENTION=0): VLLM_METAL_MEMORY_FRACTION must be auto.
• Paged KV path (VLLM_METAL_USE_PAGED_ATTENTION=1): VLLM_METAL_MEMORY_FRACTION can be auto or a numeric fraction in (0, 1].
• For paged KV with VLLM_METAL_MEMORY_FRACTION=auto, vllm-metal uses vLLM's --gpu-memory-utilization value.

VLLM_METAL_MEMORY_FRACTION	VLLM_METAL_USE_PAGED_ATTENTION	Valid?	Notes
auto	0	Yes	MLX path
auto	1	Yes	Paged KV path (default); uses --gpu-memory-utilization
0.7	1	Yes	Paged KV path with explicit memory budget
0.7	0	No	Explicit fraction without paged KV is invalid