优化与调优

优化与调优#

本指南旨在帮助用户提升 vllm-ascend 在系统层面的性能。它包括操作系统配置、库优化、部署指南等。欢迎任何反馈。

准备#

运行容器:

# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the cann base image
export IMAGE=m.daocloud.io/quay.io/ascend/cann:8.2.rc1-910b-ubuntu22.04-py3.11
docker run --rm \
--name performance-test \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash

配置您的环境:

# Configure the mirror
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" > /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list

# Install os packages
apt update && apt install wget gcc g++ libnuma-dev git vim -y

安装 vllm 和 vllm-ascend:

# Install necessary dependencies
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install modelscope pandas datasets gevent sacrebleu rouge_score pybind11 pytest

# Configure this var to speed up model download
VLLM_USE_MODELSCOPE=true

请遵循安装指南以确保 vllm、vllm-ascend 安装正确

备注

确保在完成 Python 配置后安装 vllm 和 vllm-ascend，因为这些包将使用当前环境中的 Python 来构建二进制文件。如果你在 1.1 章节之前安装了 vllm 和 vllm-ascend，二进制文件将不会使用优化的 Python。

优化#

1. Compilation Optimization#

1.1. 安装优化的 python#

从 3.6 版本开始，Python 支持 LTO 和 PGO 优化，这些优化可以在编译时启用。为了方便用户，我们直接提供了优化的 python 包。你也可以根据你的具体场景按照这个教程重新构建 python 。

mkdir -p /workspace/tmp
cd /workspace/tmp

# Download prebuilt lib and packages
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libcrypto.so.1.1
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libomp.so
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libssl.so.1.1
wget https://repo.oepkgs.net/ascend/pytorch/vllm/python/py311_bisheng.tar.gz

# Configure python and pip
cp ./*.so* /usr/local/lib
tar -zxvf ./py311_bisheng.*  -C /usr/local/
mv  /usr/local/py311_bisheng/  /usr/local/python
sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3
sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3.11
ln -sf  /usr/local/python/bin/python3  /usr/bin/python
ln -sf  /usr/local/python/bin/python3  /usr/bin/python3
ln -sf  /usr/local/python/bin/python3.11  /usr/bin/python3.11
ln -sf  /usr/local/python/bin/pip3  /usr/bin/pip3
ln -sf  /usr/local/python/bin/pip3  /usr/bin/pip

export PATH=/usr/bin:/usr/local/python/bin:$PATH

2. OS Optimization#

2.1. jemalloc#

jemalloc 是一个内存分配器，可提升多线程场景下的性能，并减少内存碎片。jemalloc 使用线程本地内存管理器来分配变量，这可以避免多线程间的锁竞争，从而大幅优化性能。

# Install jemalloc
sudo apt update
sudo apt install libjemalloc2

# Configure jemalloc
export LD_PRELOAD=/usr/lib/"$(uname -i)"-linux-gnu/libjemalloc.so.2 $LD_PRELOAD

2.2. Tcmalloc#

Tcmalloc（线程计数内存分配器）是一种通用内存分配器，通过引入多级缓存结构，在确保低延迟的同时提升整体性能，减少互斥锁竞争并优化大对象处理流程。更多详情请见此处。

# Install tcmalloc
sudo apt update
sudo apt install libgoogle-perftools4 libgoogle-perftools-dev

# Get the location of libtcmalloc.so*
find /usr -name libtcmalloc.so*

# Make the priority of tcmalloc higher
# The <path> is the location of libtcmalloc.so we get from the upper command
# Example: "$LD_PRELOAD:/usr/lib/aarch64-linux-gnu/libtcmalloc.so"
export LD_PRELOAD="$LD_PRELOAD:<path>"

# Verify your configuration
# The path of libtcmalloc.so will be contained in the result if your configuration is valid
ldd `which python`

3. torch_npu Optimization#

torch_npu 中的一些性能调优功能由环境变量控制。以下是一些功能和它们相关的环境变量。

内存优化:

# Upper limit of memory block splitting allowed (MB), Setting this parameter can prevent large memory blocks from being split.
export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:250"

# When operators on the communication stream have dependencies, they all need to be ended before being released for reuse. The logic of multi-stream reuse is to release the memory on the communication stream in advance so that the computing stream can be reused.
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

调度优化:

# Optimize operator delivery queue, this will affect the memory peak value, and may degrade if the memory is tight.
export TASK_QUEUE_ENABLE=2

# This will greatly improve the CPU bottleneck model and ensure the same performance for the NPU bottleneck model.
export CPU_AFFINITY_CONF=1

4. CANN Optimization#

4.1. HCCL 优化#

HCCL 中也有一些性能调优功能，这些功能由环境变量控制。

您可以通过设置以下环境变量来配置 HCCL 使用"AIV"模式以优化性能。在"AIV"模式下，通信由 AI 向量核心直接通过 ROCE 调度，而不是由 AI CPU 调度。

export HCCL_OP_EXPANSION_MODE="AIV"

此外，还有更多针对特定场景的性能优化功能，如下所示。

  • HCCL_INTRA_ROCE_ENABLE : 将两个 8P 之间的网格互连链路从 SDMA 链路改为 RDMA 链路，更多详情请见此处。

  • HCCL_RDMA_TC : 使用此变量配置 RDMA 网络卡的流量类别，更多详情请见此处。

  • HCCL_RDMA_SL : 使用此变量配置 RDMA 网络卡的级别服务，更多详情请查看此处。

  • HCCL_BUFFSIZE : 使用此变量控制两个 NPU 之间共享数据的缓存大小，更多详情请查看此处。

目录