动态批处理¶

动态批处理是一种在分块预填充策略中，根据资源状况和服务等级目标，动态调整每次推理迭代块大小的技术，旨在提高有效吞吐量并降低令牌间延迟。

Dynamic batch is controlled by the value of the --SLO_limits_for_dynamic_batch. Notably, only Atlas A2 inference products are supported with decode token number scales below 2048 so far. Especially, the improvements are quite obvious on Qwen, Llama models. We are working on further improvements and this feature will support more XPUs in the future.

快速入门¶

前提条件¶

动态批处理目前依赖于一个存储在查找表中的离线开销模型来优化令牌预算。该查找表保存在一个'.csv'文件中，需要先从A2-B3-BLK128.csv下载，重命名后保存到路径vllm_ascend/core/profile_table.csv。
加载查找表需要Pandas库，请确保已安装。
```
pip install pandas 
```

Tuning Parameters¶

--SLO_limits_for_dynamic_batch是动态批处理功能的调优参数（整数类型），值越大对延迟限制的约束越宽松，从而带来更高的有效吞吐量。可以根据具体模型或服务需求选择该参数。

--SLO_limits_for_dynamic_batch = -1  # Default value; dynamic batching is disabled.
--SLO_limits_for_dynamic_batch = 0  # Baseline value for dynamic batching; dynamic batching is disabled. FCFS and decode-first chunked prefilling strategy is used.
--SLO_limits_for_dynamic_batch > 0  # User-defined positive value; dynamic batching is enabled. FCFS and decode-first chunked prefilling strategy is used.

支持的模型¶

目前，动态批处理在tensor_parallel_size=8的多个稠密模型上表现更好，包括Qwen和Llama（从8B到32B）。对于不同的模型，需要设置合适的SLO_limits_for_dynamic_batch参数。该参数的经验值通常为35, 50, or 75。因此，需要进行一些额外测试以选择最佳参数。

用法¶

动态批处理用于在线推理。一个完整的可执行示例如下：

SLO_LIMIT=50
vllm serve Qwen/Qwen2.5-14B-Instruct\
    --additional_config '{"SLO_limits_for_dynamic_batch":'${SLO_LIMIT}'}' \
    --max-num-seqs 256 \
    --block-size 128 \
    --tensor_parallel_size 8 \
    --load_format dummy \
    --max_num_batched_tokens 1024 \
    --max-model-len 9000 \
    --host localhost \
    --port 12091 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code