推测解码指南

本指南介绍如何在 vLLM Ascend 中使用推测解码。推测解码是一种技术，用于改善受内存带宽限制的 LLM 推理中的 token 间延迟。

概述

vLLM Ascend 通过**提议器-验证器**架构实现推测解码：

1. 提议器 (vllm_ascend/spec_decode/)：使用多种方法生成草稿（推测）token——从简单的 n-gram 匹配到基于神经网络的草稿模型。
2. 拒绝采样器 (vllm_ascend/sample/)：根据目标模型的输出验证草稿 token，接受匹配项并拒绝不匹配项，可选优化包括块验证和熵验证。

支持以下推测解码方法：

方法	描述
ngram	匹配提示词中的 n-gram
suffix	基于后缀的模式匹配（需要 Arctic Inference）
medusa	嵌入在目标模型中的 Medusa 头
eagle	基于 EAGLE 的草稿模型
eagle3	基于 EAGLE-3 的草稿模型
mtp	使用共享嵌入头的多 token 预测
dflash	使用交叉注意力的 Draft-and-Flash
draft_model	通用外部草稿 LLM
extract_hidden_states	提取隐藏状态用于 EAGLE 训练

通用配置

所有推测解码方法都通过初始化模型或启动服务器时的 speculative_config 参数进行配置：

• method (str, 必需)：推测解码方法。必须是上表中列出的受支持方法名称之一。
• num_speculative_tokens (int, 必需)：每次前向传播生成的推测 token 数量。当可用时，从草稿模型的 n_predict 配置（例如 MTP）或 suffix_decoding_max_tree_depth（后缀方法）自动填充。
• model (str, 可选)：草稿模型的路径或 HF 仓库 ID。对于 eagle、eagle3、dflash、medusa 和 draft_model 是必需的。对于 mtp（重用目标模型）、ngram、suffix 和 extract_hidden_states 会自动解析。
• draft_tensor_parallel_size (int, 可选)：草稿模型的张量并行大小。只能为 1 或与目标模型的张量并行大小相同。
• disable_padded_drafter_batch (bool, 默认值: False)：禁用推测解码的输入填充。如果设置为 True，推测输入批次可以包含不同长度的序列，这可能仅受某些注意力后端支持。注意： 仅对 eagle、eagle3、mtp、dflash、draft_model 和 extract_hidden_states 方法有效。

离线推理 — 将 speculative_config 作为 Python 字典传递给 LLM()：

from vllm import LLM

llm = LLM(
    model="path/to/target/model",
    speculative_config={
        "method": "eagle3",
        "model": "path/to/draft/model",
        "num_speculative_tokens": 3,
    },
)

在线服务 — 将 --speculative-config（或 -sc）作为 JSON 字符串传递：

vllm serve path/to/target/model \
  --speculative-config '{"method": "eagle3", "model": "path/to/draft/model", "num_speculative_tokens": 3}'

[!NOTE] On Ascend NPUs, the npu_fused_infer_attention_score operator supports a maximum of 16 tokens per decode round. Therefore, (num_speculative_tokens + 1) must be ≤ 15.

通过匹配提示词中的 n-gram 进行推测

以下代码配置 vLLM Ascend 使用推测解码，其中候选 token 通过匹配提示词中的 n-gram 生成。

• 离线推理

  from vllm import LLM, SamplingParams
  
  prompts = [
      "The future of AI is",
  ]
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
  
  llm = LLM(
      model="meta-llama/Meta-Llama-3.1-8B-Instruct",
      tensor_parallel_size=1,
      speculative_config={
          "method": "ngram",
          "num_speculative_tokens": 5,
          "prompt_lookup_max": 4,
      },
  )
  outputs = llm.generate(prompts, sampling_params)
  
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  

Speculating using EAGLE based draft models

The following code configures vLLM Ascend to use speculative decoding where proposals are generated by an EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) based draft model.

In v0.12.0rc1 of vLLM Ascend, the async scheduler is more stable and ready to be enabled. We have adapted it to support EAGLE, and you can use it by setting async_scheduling=True as follows. If you encounter any issues, please feel free to open an issue on GitHub. As a workaround, you can disable this feature by unsetting async_scheduling=True when initializing the model.

• Offline inference

  from vllm import LLM, SamplingParams
  
  prompts = [
      "The future of AI is",
  ]
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
  
  llm = LLM(
      model="meta-llama/Meta-Llama-3.1-8B-Instruct",
      tensor_parallel_size=4,
      distributed_executor_backend="mp",
      enforce_eager=True,
      async_scheduling=True,
      speculative_config={
          "method": "eagle",
          "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B",
          "draft_tensor_parallel_size": 1,
          "num_speculative_tokens": 2,
      },
  )
  
  outputs = llm.generate(prompts, sampling_params)
  
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  

A few important things to consider when using the EAGLE based draft models:

1. The EAGLE draft models available in the HF repository for EAGLE models should be loaded and used directly by vLLM. This functionality was added in PR #4893. If you are using a vLLM version released before this pull request was merged, please update to a more recent version.

2. The EAGLE based draft models need to be run without tensor parallelism (i.e. draft_tensor_parallel_size is set to 1 in speculative_config), although it is possible to run the main model using tensor parallelism (see example above).

3. When using EAGLE-3 based draft model, option "method" must be set to "eagle3". That is, to specify "method": "eagle3" in speculative_config.

4. After enabling EAGLE, the main model needs to verify (1 + K) tokens generated by the main model and the draft model in one decoding process. And the fullgraph mode will fix the number of tokens during the verification stage, so cudagraph_capture_sizes must be a list of capture sizes, where each size is calculated as n * (K + 1) for each batch size n you want to support. For instance, to support batch sizes from 1 to 4 with num_speculative_tokens = 4, cudagraph_capture_sizes should be set to [5, 10, 15, 20].

Speculating using MTP

MTP (Multi-Token Prediction) boosts inference performance by parallelizing the prediction of multiple tokens, shifting from single-token to multi-token generation. This approach significantly increases generation throughput and achieves multiplicative acceleration in inference speed — all without compromising output quality.

• Online inference

  vllm serve /deepseek-ai/DeepSeek-V3.2-Exp-W8A8 \
  --port 20004 \
  --data-parallel-size 1 \
  --tensor-parallel-size 16 \
  --enable-expert-parallel \
  --seed 1024 \
  --served-model-name dsv3 \
  --max-model-len 36768 \
  --max-num-batched-tokens 5000 \
  --max-num-seqs 10 \
  --quantization ascend \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --speculative-config '{"num_speculative_tokens": 2, "method":"mtp", "disable_padded_drafter_batch": false}'
  

[!NOTE] Due to the fact that only a single layer of weights is exposed in DeepSeek's MTP, accuracy and performance are not effectively guaranteed in scenarios where num_speculative_tokens > 1 (especially ≥ 3).

In the fullgraph mode with num_speculative_tokens > 1, the capture size of each ACLGraph must be an integer multiple of (num_speculative_tokens + 1).

Speculating using Suffix Decoding

The following code configures vLLM to use speculative decoding where proposals are generated using Suffix Decoding (SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications).

Like n-gram, Suffix Decoding can generate draft tokens by pattern-matching using the last n generated tokens. Unlike n-gram, Suffix Decoding (1) can pattern-match against both the prompt and previous generations, (2) uses frequency counts to propose the most likely continuations, and (3) speculates an adaptive number of tokens for each request at each iteration to get better acceptance rates.

Suffix Decoding can achieve better performance for tasks with high repetition, such as code-editing, agentic loops (e.g. self-reflection, self-consistency), and RL rollouts.

[!NOTE] Suffix Decoding requires Arctic Inference. You can install it with pip install arctic-inference.

• Offline inference

```python from vllm import LLM, SamplingParams

prompts = [
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,
    enforce_eager=True,
    speculative_config={
        "method": "suffix",
        "num_speculative_tokens": 15,
    },
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```

Extracting Hidden States

The extract_hidden_states method is a special speculative decoding mode that does not perform actual speculation. Instead, it extracts hidden states from specified layers of the target model and saves them to disk. This is primarily used for collecting training data for EAGLE-style draft models.

[!NOTE] This method produces only 1 output token per request. The primary output is the hidden states saved to disk, not the generated text.

• Offline inference

  import tempfile
  
  from safetensors import safe_open
  from vllm import LLM, SamplingParams
  
  def main():
      with tempfile.TemporaryDirectory() as tmpdirname:
          llm = LLM(
              model="Qwen/Qwen3-8B",
              tensor_parallel_size=1,
              speculative_config={
                  "method": "extract_hidden_states",
                  "num_speculative_tokens": 1,
                  "draft_model_config": {
                      "hf_config": {
                          # Layer indices to extract hidden states from
                          "eagle_aux_hidden_state_layer_ids": [2, 18, 34],
                      }
                  },
              },
              kv_transfer_config={
                  "kv_connector": "ExampleHiddenStatesConnector",
                  "kv_role": "kv_producer",
                  "kv_connector_extra_config": {
                      "shared_storage_path": tmpdirname,
                  },
              },
          )
  
          prompts = ["Hello, how are you?", "What is machine learning?"]
          sampling_params = SamplingParams(max_tokens=1)
          outputs = llm.generate(prompts, sampling_params)
  
          for output in outputs:
              print("Prompt:", output.prompt)
              print("Prompt token ids:", output.prompt_token_ids)
  
              hidden_states_path = output.kv_transfer_params.get("hidden_states_path")
              print("Hidden states saved to:", hidden_states_path)
  
              with safe_open(hidden_states_path, "pt") as f:
                  token_ids = f.get_tensor("token_ids")
                  hidden_states = f.get_tensor("hidden_states")
                  print("Shape:", hidden_states.shape)
                  # Shape: (num_tokens, num_layers, hidden_size)
  
  if __name__ == "__main__":
      main()
  

Key configuration parameters:

1. num_speculative_tokens: Must be set to 1. This method does not perform actual speculation, so the value is fixed.

2. eagle_aux_hidden_state_layer_ids: List of layer indices from which to extract hidden states. For example, [2, 18, 34] extracts from layers 2, 18, and 34.

3. kv_connector: Must be set to "ExampleHiddenStatesConnector" to enable saving hidden states to disk.

4. kv_role: Must be set to "kv_producer" for the extraction mode.

5. shared_storage_path: Directory where hidden states will be saved as .safetensors files (one per request).

Block Verify and Entropy Verify

vLLM Ascend provides two optional optimizations for the rejection sampler in speculative decoding: Block Verify and Entropy Verify. These features trade a small amount of output precision for improved inference throughput.

[!WARNING] Both Block Verify and Entropy Verify modify the token acceptance criteria and may cause minor precision degradation (e.g., slightly different output tokens compared to the standard rejection sampler). Evaluate the quality impact on your specific workload before enabling them in production.

Block Verify

Block Verify evaluates all draft tokens as a block using cumulative probability products, rather than checking each token independently. This can improve the acceptance rate and reduce the overhead of rejection sampling, especially when num_speculative_tokens >= 3.

Entropy Verify

Entropy Verify adjusts the acceptance threshold based on the entropy of the target distribution:

• High entropy (uncertain distribution) → lower effective threshold → more tokens accepted
• Low entropy (confident distribution) → higher effective threshold → stricter rejection

This entropy-aware threshold is controlled by two parameters:

• posterior_threshold (default: 0.95, range: (0, 1]): The upper bound of the modified threshold. Even when entropy is very low, the effective threshold will not exceed this value.
• posterior_alpha (default: 0.4, range: >= 0): Controls how strongly entropy influences the threshold. A higher alpha makes the threshold more sensitive to entropy changes, resulting in a higher acceptance rate for speculative tokens but also greater precision loss. You need to tune this value based on your specific model and dataset. When alpha is 0, entropy has no effect and the threshold equals posterior_threshold.

Usage

• Online inference

  vllm serve <model> --additional-config \
      '{"rejection_sampler_config": {"enable_block_verify": true, \
      "enable_entropy_verify": true, "posterior_threshold": 0.95, \
      "posterior_alpha": 0.4}}'
  

• Offline inference

  llm = LLM(
      model,
      additional_config={
          "rejection_sampler_config": {
              "enable_block_verify": True,
              "enable_entropy_verify": True,
              "posterior_threshold": 0.95,
              "posterior_alpha": 0.4,
          }
      },
  )
  

这两个功能可以独立启用，也可以同时启用。同时使用时，块验证的累积接受率将与熵验证的熵调整阈值相结合。