Speculative Decoding Guide

This guide shows how to use Speculative Decoding with vLLM Ascend. Speculative decoding is a technique which improves inter-token latency in memory-bound LLM inference.

Overview

vLLM Ascend implements speculative decoding through a proposer-verifier architecture:

1. Proposer (vllm_ascend/spec_decode/): Generates draft (speculative) tokens using various methods — from simple n-gram matching to neural-network-based draft models.
2. Rejection Sampler (vllm_ascend/sample/): Verifies draft tokens against the target model's output, accepting matches and rejecting mismatches, with optional optimizations including Block Verify and Entropy Verify.

The following speculative decoding methods are supported:

Method	Description
ngram	Match n-grams from the prompt
suffix	Suffix-based pattern matching (requires Arctic Inference)
medusa	Medusa heads embedded in the target model
eagle	EAGLE-based draft model
eagle3	EAGLE-3 based draft model
mtp	Multi-Token Prediction with shared embedding head
dflash	Block diffusion-based parallel draft model
dspark	Semi-autoregressive block drafting with a sequential Markov logit-bias head
draft_model	Generic external draft LLM
extract_hidden_states	Extract hidden states for EAGLE training

Common Configuration

All speculative decoding methods are configured through the speculative_config parameter when initializing the model or starting the server:

• method (str, required): The speculative decoding method. Must be one of the supported method names listed in the table above.
• num_speculative_tokens (int, required): Number of speculative tokens to generate per forward pass. Auto-filled from the draft model's n_predict config (e.g., MTP) or suffix_decoding_max_tree_depth (suffix method) when available. > Note: For PD Separation deployment, num_speculative_tokens should be subject to one of the following conditions: > > 1. Hybrid Mamba models (e.g., Qwen-Next and Qwen3.5 series): num_speculative_tokens should be equal on P nodes and D nodes. > 2. Other models: num_speculative_tokens on P nodes should be 1, and num_speculative_tokens on D nodes should be greater or equal to 1.
• model (str, optional): Path or HF repo ID for the draft model. Required for eagle, eagle3, dflash, medusa, and draft_model. Automatically resolved for mtp (reuses target model), ngram, suffix, and extract_hidden_states.
• draft_tensor_parallel_size (int, optional): Tensor parallelism size for the draft model. Can only be 1 or the same as the target model's tensor parallel size.
• disable_padded_drafter_batch (bool, default: False): Disable input padding for speculative decoding. If set to True, speculative input batches can contain sequences of different lengths, which may only be supported by certain attention backends. Note: Only effective with eagle, eagle3, mtp, dflash, draft_model, and extract_hidden_states methods.

Offline inference — pass speculative_config as a Python dict to LLM():

from vllm import LLM

llm = LLM(
    model="path/to/target/model",
    speculative_config={
        "method": "eagle3",
        "model": "path/to/draft/model",
        "num_speculative_tokens": 3,
    },
)

Online serving — pass --speculative-config (or -sc) as a JSON string:

vllm serve path/to/target/model \
  --speculative-config '{"method": "eagle3", "model": "path/to/draft/model", "num_speculative_tokens": 3}'

[!NOTE] On Ascend NPUs, the npu_fused_infer_attention_score operator supports a maximum of 16 tokens per decode round. Therefore, (num_speculative_tokens + 1) must be ≤ 16.

Speculating by matching n-grams in the prompt

The following code configures vLLM Ascend to use speculative decoding where proposals are generated by matching n-grams in the prompt.

• Offline inference

  from vllm import LLM, SamplingParams
  
  prompts = [
      "The future of AI is",
  ]
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
  
  llm = LLM(
      model="meta-llama/Meta-Llama-3.1-8B-Instruct",
      tensor_parallel_size=1,
      speculative_config={
          "method": "ngram",
          "num_speculative_tokens": 5,
          "prompt_lookup_max": 4,
      },
  )
  outputs = llm.generate(prompts, sampling_params)
  
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  

Speculating using EAGLE based draft models

The following code configures vLLM Ascend to use speculative decoding where proposals are generated by an EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) based draft model.

• Offline inference

  from vllm import LLM, SamplingParams
  
  prompts = [
      "The future of AI is",
  ]
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
  
  llm = LLM(
      model="meta-llama/Meta-Llama-3.1-8B-Instruct",
      tensor_parallel_size=4,
      distributed_executor_backend="mp",
      enforce_eager=True,
      speculative_config={
          "method": "eagle",
          "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B",
          "draft_tensor_parallel_size": 1,
          "num_speculative_tokens": 2,
      },
  )
  
  outputs = llm.generate(prompts, sampling_params)
  
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  

• Online inference

  vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --tensor-parallel-size 4 \
    --enforce-eager \
    --speculative-config '{"method": "eagle", "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B", "draft_tensor_parallel_size": 1, "num_speculative_tokens": 2}'
  

A few important things to consider when using the EAGLE based draft models:

1. The EAGLE draft models available in the HF repository for EAGLE models should be loaded and used directly by vLLM. This functionality was added in PR #4893. If you are using a vLLM version released before this pull request was merged, please update to a more recent version.

2. The EAGLE based draft models need to be run without tensor parallelism (i.e. draft_tensor_parallel_size is set to 1 in speculative_config), although it is possible to run the main model using tensor parallelism (see example above).

3. When using EAGLE-3 based draft model, option "method" must be set to "eagle3". That is, to specify "method": "eagle3" in speculative_config.

4. After enabling EAGLE, the main model needs to verify (1 + K) tokens generated by the main model and the draft model in one decoding process. And the fullgraph mode will fix the number of tokens during the verification stage, so cudagraph_capture_sizes must be a list of capture sizes, where each size is calculated as n * (K + 1) for each batch size n you want to support. For instance, to support batch sizes from 1 to 4 with num_speculative_tokens = 4, cudagraph_capture_sizes should be set to [5, 10, 15, 20].

Speculating using MTP

MTP (Multi-Token Prediction) boosts inference performance by parallelizing the prediction of multiple tokens, shifting from single-token to multi-token generation. This approach significantly increases generation throughput and achieves multiplicative acceleration in inference speed — all without compromising output quality.

• Offline inference

  from vllm import LLM, SamplingParams
  
  prompts = [
      "The future of AI is",
  ]
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
  
  llm = LLM(
      model="deepseek-ai/DeepSeek-V3.2-Exp-W8A8",
      tensor_parallel_size=16,
      enable_expert_parallel=True,
      max_model_len=36768,
      max_num_seqs=10,
      quantization="ascend",
      trust_remote_code=True,
      gpu_memory_utilization=0.9,
      compilation_config={"cudagraph_mode": "FULL_DECODE_ONLY"},
      speculative_config={
          "method": "mtp",
          "num_speculative_tokens": 2,
          "disable_padded_drafter_batch": False,
      },
  )
  outputs = llm.generate(prompts, sampling_params)
  
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  

• Online inference

  vllm serve /deepseek-ai/DeepSeek-V3.2-Exp-W8A8 \
  --port 20004 \
  --data-parallel-size 1 \
  --tensor-parallel-size 16 \
  --enable-expert-parallel \
  --seed 1024 \
  --served-model-name dsv3 \
  --max-model-len 36768 \
  --max-num-batched-tokens 5000 \
  --max-num-seqs 10 \
  --quantization ascend \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --speculative-config '{"num_speculative_tokens": 2, "method":"mtp", "disable_padded_drafter_batch": false}'
  

[!NOTE] Due to the fact that only a single layer of weights is exposed in DeepSeek's MTP, accuracy and performance are not effectively guaranteed in scenarios where num_speculative_tokens > 1 (especially ≥ 3).

In the fullgraph mode with num_speculative_tokens > 1, the capture size of each ACLGraph must be an integer multiple of (num_speculative_tokens + 1).

Speculating using DFlash

The following code configures vLLM Ascend to use speculative decoding where proposals are generated by a DFlash block diffusion-based parallel draft model.

• Offline inference

  from vllm import LLM, SamplingParams
  from vllm.config import CompilationConfig
  
  prompts = [
      "The future of AI is",
  ]
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
  
  llm = LLM(
      model="Qwen/Qwen3-8B",
      tensor_parallel_size=1,
      distributed_executor_backend="mp",
      max_model_len=4096,
      max_num_seqs=16,
      gpu_memory_utilization=0.8,
      enable_prefix_caching=False,
      speculative_config={
          "method": "dflash",
          "model": "z-lab/Qwen3-8B-DFlash-b16",
          "num_speculative_tokens": 7,
      },
      compilation_config=CompilationConfig(cudagraph_mode="FULL_DECODE_ONLY"),
  )
  outputs = llm.generate(prompts, sampling_params)
  
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  

• Online inference

  vllm serve Qwen/Qwen3-8B \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --max-num-seqs 256 \
    --gpu-memory-utilization 0.8 \
    --no-enable-prefix-caching \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3-8B-DFlash-b16", "num_speculative_tokens": 7}'
  

Speculating using DSpark

The following code configures vLLM Ascend to use speculative decoding where proposals are generated by a DSpark semi-autoregressive block drafter with a sequential Markov logit-bias head.

• Offline inference

  from vllm import LLM, SamplingParams
  from vllm.config import CompilationConfig
  
  prompts = [
      "The future of AI is",
  ]
  sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
  
  llm = LLM(
      model="Qwen/Qwen3-8B",
      tensor_parallel_size=1,
      distributed_executor_backend="mp",
      max_model_len=4096,
      max_num_seqs=8,
      gpu_memory_utilization=0.8,
      enable_prefix_caching=False,
      speculative_config={
          "method": "dspark",
          "model": "deepseek-ai/dspark_qwen3_8b_block7",
          "num_speculative_tokens": 7,
          "enforce_eager": True,
      },
      compilation_config=CompilationConfig(cudagraph_mode="FULL_DECODE_ONLY"),
  )
  outputs = llm.generate(prompts, sampling_params)
  
  for output in outputs:
      prompt = output.prompt
      generated_text = output.outputs[0].text
      print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
  

• Online inference

  vllm serve Qwen/Qwen3-8B \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --max-num-seqs 8 \
    --gpu-memory-utilization 0.8 \
    --no-enable-prefix-caching \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --speculative-config '{"method": "dspark", "model": "deepseek-ai/dspark_qwen3_8b_block7", "num_speculative_tokens": 7, "enforce_eager": true}'
  

Speculating using Suffix Decoding

The following code configures vLLM to use speculative decoding where proposals are generated using Suffix Decoding (SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications).

Like n-gram, Suffix Decoding can generate draft tokens by pattern-matching using the last n generated tokens. Unlike n-gram, Suffix Decoding (1) can pattern-match against both the prompt and previous generations, (2) uses frequency counts to propose the most likely continuations, and (3) speculates an adaptive number of tokens for each request at each iteration to get better acceptance rates.

Suffix Decoding can achieve better performance for tasks with high repetition, such as code-editing, agentic loops (e.g. self-reflection, self-consistency), and RL rollouts.

[!NOTE] Suffix Decoding requires Arctic Inference. You can install it with pip install arctic-inference.

• Offline inference

```python from vllm import LLM, SamplingParams

prompts = [
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,
    enforce_eager=True,
    speculative_config={
        "method": "suffix",
        "num_speculative_tokens": 15,
    },
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```

• Online inference

  vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --tensor-parallel-size 1 \
    --enforce-eager \
    --speculative-config '{"method": "suffix", "num_speculative_tokens": 15}'
  

Extracting Hidden States

The extract_hidden_states method is a special speculative decoding mode that does not perform actual speculation. Instead, it extracts hidden states from specified layers of the target model and saves them to disk. This is primarily used for collecting training data for EAGLE-style draft models.

[!NOTE] This method produces only 1 output token per request. The primary output is the hidden states saved to disk, not the generated text.

• Offline inference

  import tempfile
  
  from safetensors import safe_open
  from vllm import LLM, SamplingParams
  
  def main():
      with tempfile.TemporaryDirectory() as tmpdir:
          llm = LLM(
              model="Qwen/Qwen3-8B",
              tensor_parallel_size=1,
              speculative_config={
                  "method": "extract_hidden_states",
                  "num_speculative_tokens": 1,
                  "draft_model_config": {
                      "hf_config": {
                          # Layer indices to extract hidden states from
                          "eagle_aux_hidden_state_layer_ids": [2, 18, 34],
                      }
                  },
              },
              kv_transfer_config={
                  "kv_connector": "ExampleHiddenStatesConnector",
                  "kv_role": "kv_producer",
                  "kv_connector_extra_config": {
                      "shared_storage_path": tmpdir,
                  },
              },
          )
  
          prompts = ["Hello, how are you?", "What is machine learning?"]
          sampling_params = SamplingParams(max_tokens=1)
          outputs = llm.generate(prompts, sampling_params)
  
          for output in outputs:
              print("Prompt:", output.prompt)
              print("Prompt token ids:", output.prompt_token_ids)
  
              hidden_states_path = output.kv_transfer_params.get("hidden_states_path")
              print("Hidden states saved to:", hidden_states_path)
  
              with safe_open(hidden_states_path, "pt") as f:
                  token_ids = f.get_tensor("token_ids")
                  hidden_states = f.get_tensor("hidden_states")
                  print("Shape:", hidden_states.shape)
                  # Shape: (num_tokens, num_layers, hidden_size)
  
  if __name__ == "__main__":
      main()
  

• Online inference

  For improved performance, it is recommended to use a RAM-mounted file system such as /dev/shm/ for online usage in which the client cleans up the files soon after they are generated.

  vllm serve Qwen/Qwen3-8B \
    --tensor-parallel-size 1 \
    --speculative-config '{"method": "extract_hidden_states", "num_speculative_tokens": 1, "draft_model_config": {"hf_config": {"eagle_aux_hidden_state_layer_ids": [2, 18, 34]}}}' \
    --kv-transfer-config '{"kv_connector": "ExampleHiddenStatesConnector", "kv_role": "kv_producer", "kv_connector_extra_config": {"shared_storage_path": "/dev/shm/hidden_states"}}'
  

Key configuration parameters:

1. num_speculative_tokens: Must be set to 1. This method does not perform actual speculation, so the value is fixed.

2. eagle_aux_hidden_state_layer_ids: List of layer indices from which to extract hidden states. For example, [2, 18, 34] extracts from layers 2, 18, and 34.

3. kv_connector: Must be set to "ExampleHiddenStatesConnector" to enable saving hidden states to disk.

4. kv_role: Must be set to "kv_producer" for the extraction mode.

5. shared_storage_path: Directory where hidden states will be saved as .safetensors files (one per request).

Dynamic Speculative Decoding

Dynamic Speculative Decoding adapts the number of draft tokens (K) at runtime, instead of always using a fixed num_speculative_tokens. This helps keep speculative decoding beneficial as concurrency and draft confidence change. vLLM Ascend currently provides two approaches:

• Confidence-based verify length (DSpark / DFlash): adjust the per-request verify length from draft confidence. DSpark uses a dedicated confidence head; DFlash is head-free and uses max(softmax(logits)) of the drafted token as a confidence proxy.
• Batch-size based (Autoregressive): select a shared K from concurrency ranges via num_speculative_tokens_per_batch_size.

Confidence-based verify length (DSpark / DFlash)

This approach adapts how many drafted tokens are verified per request. It can reduce verify cost when the drafter is less confident about later tokens, while still allowing longer speculation when confidence is high.

[!NOTE] This is an exploratory feature and currently targets model runner v1 only. Supported dynamic methods are "dspark" (confidence head) and "dflash" (head-free). Among DSpark draft models, only the Qwen series is supported today; support for models such as GLM and DeepSeek is being added incrementally.

How it works

When dynamic_spec_config.method is set to "dspark" or "dflash", the corresponding proposer:

1. Runs the draft model as usual to produce up to num_speculative_tokens draft tokens.
2. Estimates per-position acceptance likelihood for each request:
3. DSpark: sigmoid output of the DSpark confidence head.
4. DFlash (head-free): max(softmax(logits)) of the argmax draft token (no extra neural head).
5. Periodically recomputes a shared per-request verify budget from those confidence scores.
6. Allocates that budget across requests (each request keeps at least min_verify_tokens), so each request verifies only a prefix of draft tokens.

The resulting per-request verify lengths are consumed by the model runner when collecting draft tokens for the target-model verify step.

Configuration

Enable this approach through additional_config.dynamic_spec_config. See Additional Configuration for the full parameter reference.

You still need a matching speculative_config (method: "dspark" or "dflash", draft model path, and num_speculative_tokens). Dynamic decoding only decides how many of those drafted tokens are verified per request. Set dynamic_spec_config.method to the same method name as in speculative_config.

Offline inference example

from vllm import LLM, SamplingParams
from vllm.config import CompilationConfig

prompts = [
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="Qwen/Qwen3-8B",
    tensor_parallel_size=1,
    distributed_executor_backend="mp",
    max_model_len=4096,
    max_num_seqs=8,
    gpu_memory_utilization=0.8,
    enable_prefix_caching=False,
    speculative_config={
        "method": "dspark",
        "model": "deepseek-ai/dspark_qwen3_8b_block7",
        "num_speculative_tokens": 7,
    },
    additional_config={
        "dynamic_spec_config": {
            "method": "dspark",
            "method_params": {
                "initial_verify_budget_per_req": 5,
                "budget_update_interval": 50,
                "budget_threshold": 0.7,
            },
        },
    },
    compilation_config=CompilationConfig(cudagraph_mode="FULL"),
)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Batch-size based (Autoregressive)

This approach selects a shared speculative length K from the current concurrency (batch size). It is intended for autoregressive draft methods such as MTP, EAGLE / EAGLE-3, DFlash, and n-gram.

Why is it needed?

SD methods need to verify K tokens for each sequence during decoding. As batch size (BS) increases, the effective batch becomes BS * K, which increases the compute requirement during verification. When BS * K goes beyond a critical batch size, speculative decoding can hurt decode speed (TPOT). Batch-size based Dynamic SD tunes K to an optimal value so that speculative decoding remains beneficial.

Use cases

• Variable concurrency on the same deployment: K decreases as concurrency increases.
• RL rollout workloads that start with high BS and later shrink to a few long-tail requests: K can increase again toward the end of the rollout.
• Currently supports MTP, EAGLE-3, DFlash, n-gram, and similar methods. Suffix Decoding is already per-request dynamic, so an outer dynamic K is redundant or conflicting; this is an upstream constraint, not Ascend-specific. DSpark and MTP + DCP + DSD are currently not supported on this path.

--speculative-config schema

To use Batch-size based Dynamic SD, add num_speculative_tokens_per_batch_size to the speculative config of a supported method. It is a list of lists, where each entry is [start_bs, end_bs, optimal_K]: when concurrency is within [start_bs, end_bs], optimal_K draft tokens are used. For example:

--speculative-config '{
    "method": "eagle",
    "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B",
    "num_speculative_tokens": 3,
    "num_speculative_tokens_per_batch_size": [
      [1, 64, 3],
      [65, 128, 1],
      [129, 512, 0]
    ]
  }'

implies that:

• K=3 will be used when the concurrency is in range [1, 64]
• K=1 will be used when the concurrency is in range [65, 128]
• K=0 will be used when the concurrency is in range [129, 512], i.e., no draft tokens will be produced.

Online examples

Dynamic SD Eagle Drafter

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{
    "method": "eagle",
    "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B",
    "num_speculative_tokens": 3,
    "num_speculative_tokens_per_batch_size": [
      [1, 64, 3],
      [65, 128, 1],
      [129, 512, 0]
    ]
  }'

Dynamic SD Eagle3 Drafter

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{
    "method": "eagle3",
    "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B",
    "num_speculative_tokens": 3,
    "num_speculative_tokens_per_batch_size": [
      [1, 16, 5],
      [17, 32, 4],
      [33, 64, 3],
      [65, 128, 1],
      [129, 512, 0]
    ]
  }'

Block Verify and Entropy Verify

vLLM Ascend provides two optional optimizations for the rejection sampler in speculative decoding: Block Verify and Entropy Verify. These features trade a small amount of output precision for improved inference throughput.

[!WARNING] Both Block Verify and Entropy Verify modify the token acceptance criteria and may cause minor precision degradation (e.g., slightly different output tokens compared to the standard rejection sampler). Evaluate the quality impact on your specific workload before enabling them in production.

Block Verify

Block Verify evaluates all draft tokens as a block using cumulative probability products, rather than checking each token independently. This can improve the acceptance rate and reduce the overhead of rejection sampling, especially when num_speculative_tokens >= 3.

Entropy Verify

Entropy Verify adjusts the acceptance threshold based on the entropy of the target distribution:

• High entropy (uncertain distribution) → lower effective threshold → more tokens accepted
• Low entropy (confident distribution) → higher effective threshold → stricter rejection

This entropy-aware threshold is controlled by two parameters:

• posterior_threshold (default: 0.95, range: (0, 1]): The upper bound of the modified threshold. Even when entropy is very low, the effective threshold will not exceed this value.
• posterior_alpha (default: 0.4, range: >= 0): Controls how strongly entropy influences the threshold. A higher alpha makes the threshold more sensitive to entropy changes, resulting in a higher acceptance rate for speculative tokens but also greater precision loss. You need to tune this value based on your specific model and dataset. When alpha is 0, entropy has no effect and the threshold equals posterior_threshold.

Usage

• Online inference

  vllm serve <model> --additional-config \
      '{"rejection_sampler_config": {"enable_block_verify": true, \
      "enable_entropy_verify": true, "posterior_threshold": 0.95, \
      "posterior_alpha": 0.4}}'
  

• Offline inference

  llm = LLM(
      model,
      additional_config={
          "rejection_sampler_config": {
              "enable_block_verify": True,
              "enable_entropy_verify": True,
              "posterior_threshold": 0.95,
              "posterior_alpha": 0.4,
          }
      },
  )
  

Both features can be enabled independently or together. When used together, the cumulative acceptance from Block Verify is combined with the entropy-adjusted threshold from Entropy Verify.

Synthetic Rejection Sampling

In addition to the default standard rejection sampling (which accepts a draft token when target_prob / draft_prob >= uniform), vLLM Ascend supports synthetic rejection sampling. In this mode each draft token at position i is accepted with a configured probability rates[i], compared against a uniform random draw — independent of the target/draft probability match. The first rejected position stops the request and emits a recovered token; if every draft token is accepted, the bonus token is appended.

Why use it

Synthetic mode decouples the verifier's acceptance logic from draft-model quality, letting you drive the speculative-decoding pipeline at a controlled, fixed acceptance rate. It is useful for:

• Benchmarking the verify path (kernel + sampler + token assembly) throughput/latency at a known acceptance rate, without depending on a well-trained drafter.
• Kernel validation — because acceptance reduces to uniform < rate, the result is fully determined by the inputs. The e2e test compares the Ascend Triton kernel (SYNTHETIC_MODE=True) against the PyTorch reference under identical inputs, so the two must agree bit-for-bit.
• Stress-testing the high-acceptance regime of the speculative pipeline.

[!WARNING] Synthetic mode accepts draft tokens regardless of whether they match the target distribution, so the generated output is not semantically correct. Use it for benchmarking and validation only, never for production serving.

How to use

Synthetic mode is configured through speculative_config, alongside the proposer settings:

• rejection_sample_method (str, default: "standard"): set to "synthetic" to enable.
• synthetic_acceptance_rates (list[float], optional): per-position acceptance rates. Must have length num_speculative_tokens, each entry in [0, 1], and be monotonically non-increasing.
• synthetic_acceptance_length (float, optional): target mean acceptance length in [1, num_speculative_tokens + 1], resolved internally to an equivalent synthetic_acceptance_rates schedule. Mutually exclusive with synthetic_acceptance_rates; exactly one must be provided when rejection_sample_method == "synthetic".

Offline inference:

from vllm import LLM

llm = LLM(
    model="path/to/target/model",
    speculative_config={
        "method": "eagle3",
        "model": "path/to/draft/model",
        "num_speculative_tokens": 3,
        "rejection_sample_method": "synthetic",
        "synthetic_acceptance_rates": [0.9, 0.6, 0.3],
    },
)

Online serving:

vllm serve path/to/target/model \
  --speculative-config '{"method": "eagle3", "model": "path/to/draft/model", \
    "num_speculative_tokens": 3, "rejection_sample_method": "synthetic", \
    "synthetic_acceptance_rates": [0.9, 0.6, 0.3]}'

Ascend-specific notes

Upstream vLLM's synthetic path draws the per-token uniform numbers with tl_rand64 (fp64) directly inside the Triton kernel. NPU Triton does not support tl_rand64/fp64, so vLLM Ascend reimplements synthetic mode in vllm_ascend/sample/rejection_sampler.py and vllm_ascend/ops/triton/reject_sample.py: the kernels receive the uniform probabilities as an explicit pointer argument instead of generating them internally. The probabilities are produced by generate_uniform_probs in fp64 (matching upstream, so the draw never samples exactly 0.0) and cast to fp32 for the greedy kernels, which the fp32-only Ascend Triton kernels then compare against rates[pos]. This is why synthetic mode is available on the Ascend v1/sample rejection-sampler path.