Speculative Decoding

vllm-metal supports three speculative decoding methods on the paged-attention path. Use vLLM's speculative decoding guide for method behavior and configuration details.

	MTP	Draft model	N-gram
--speculative-config method	mtp	draft_model	ngram
Target models	Gemma4	Non-hybrid paged-attention models	Non-hybrid paged-attention models
Draft source	Matching Gemma4 assistant checkpoint	Separate smaller model	Prompt and output token history
num_speculative_tokens	Configurable (2–3 typical)	Configurable (3–5 typical)	Configurable (3–5 typical)
Additional model weights	Assistant checkpoint	Draft model	None
Additional KV cache	None; reads target KV	Second scheduler-managed cache	None

All three methods currently have these Metal-specific constraints:

• Only greedy requests (temperature=0) are drafted. Other requests run without speculation.
• Scheduling must be synchronous. The Metal platform disables async scheduling when speculative decoding is configured.
• Pipeline parallelism is not supported with speculative decoding.
• Hybrid GDN targets and heterogeneous draft vocabularies are not supported.
• long_prefill_token_threshold, when set, must be at least 1 + num_speculative_tokens.

Gemma4 MTP

Follow the upstream MTP guide for Gemma4 assistant behavior. Use matching target and assistant families:

Target	Assistant
Gemma4 E2B-it	Gemma4 E2B-it assistant bf16
Gemma4 E4B-it	Gemma4 E4B-it assistant bf16
Gemma4 31B-it bf16	Gemma4 31B-it assistant bf16

Start with num_speculative_tokens=3. On the measured E4B workload, higher values improved single-stream throughput but reduced saturated throughput. Benchmark the intended batch shape before changing it.

Example

export TARGET=/path/to/gemma-4-E2B-it
export ASSISTANT=/path/to/gemma-4-E2B-it-assistant-bf16

VLLM_METAL_MEMORY_FRACTION=0.5 \
  vllm serve "$TARGET" \
    --max-model-len 1024 \
    --max-num-batched-tokens 1024 \
    --max-num-seqs 4 \
    --no-async-scheduling \
    --speculative-config "{\"method\":\"mtp\",\"model\":\"$ASSISTANT\",\"num_speculative_tokens\":3}"

Remote Hugging Face checkpoints are supported. Pin revision in speculative_config when publishing benchmark results.

Draft model

Follow the upstream draft-model guide for configuration details. The draft must use the target vocabulary and full attention. Sliding-window and hybrid draft models are rejected at startup. Its committed KV cache shares the Metal KV memory budget with the target.

Example

VLLM_METAL_MEMORY_FRACTION=0.55 \
  vllm serve Qwen/Qwen3-8B \
    --max-model-len 2048 \
    --no-async-scheduling \
    --speculative-config '{"method":"draft_model","model":"Qwen/Qwen3-0.6B","num_speculative_tokens":3}'

N-gram

Follow the upstream N-gram guide for configuration details. N-gram speculation needs no additional model or KV cache. Its benefit depends on repeated token spans in the request history.

Example

VLLM_METAL_USE_PAGED_ATTENTION=1 \
  vllm serve Qwen/Qwen3-8B \
    --max-model-len 2048 \
    --no-async-scheduling \
    --speculative-config '{"method":"ngram","num_speculative_tokens":3,"prompt_lookup_min":2,"prompt_lookup_max":3}'

Benchmarking

Use vLLM's benchmark CLI for serving workloads. For a reproducible Gemma4 target-only versus MTP comparison, use the in-tree benchmark:

python -m tools.benchmark.gemma4_mtp_benchmark --help

tools/README.md documents the before-and-after commands and the natural-prompt dataset used for speculative-decoding measurements.