使用 AISBench

本文档指导您如何使用 AISBench 进行准确率测试。AISBench 为多种数据集提供准确率和性能评估。

在线服务器模式

1.启动 vLLM 服务器

您可以通过运行 Docker 容器，在单个 NPU 上启动 vLLM 服务器：

# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci7
# Update the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-e VLLM_USE_MODELSCOPE=True \
-e PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256 \
-it $IMAGE \
/bin/bash

在 Docker 容器中运行 vLLM 服务器。

vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 35000 &

Note

--max-model-len 应设置为大于 35000，这适用于大多数数据集。否则可能会影响准确率评估结果。

如果您看到如下日志，表明 vLLM 服务器已成功启动：

INFO:     Started server process [9446]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

2.使用 AISBench 运行不同数据集

安装 AISBench

Refer to AISBench for details. 安装 AISBench from source.

git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517

安装 AISBench 的额外依赖。

pip3 install -r requirements/api.txt
pip3 install -r requirements/extra.txt

运行 ais_bench -h 命令检查安装是否成功。

下载数据集

您可以选择一个或多个数据集进行准确率评估。

1. C-Eval 数据集。

   以 C-Eval 数据集为例。您可以参考 数据集列表 查看更多数据集信息。每个数据集都有一个 README.md 文件，提供了详细的下载和安装说明。

   下载数据集并将其安装到指定路径。

   cd ais_bench/datasets
   mkdir ceval/
   mkdir ceval/formal_ceval
   cd ceval/formal_ceval
   wget https://www.modelscope.cn/datasets/opencompass/ceval-exam/resolve/master/ceval-exam.zip
   unzip ceval-exam.zip
   rm ceval-exam.zip
   

2. MMLU dataset.

   cd ais_bench/datasets
   wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/mmlu.zip
   unzip mmlu.zip
   rm mmlu.zip
   

3. GPQA dataset.

   cd ais_bench/datasets
   wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gpqa.zip
   unzip gpqa.zip
   rm gpqa.zip
   

4. MATH dataset.

   cd ais_bench/datasets
   wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/math.zip
   unzip math.zip
   rm math.zip
   

5. LiveCodeBench dataset.

   cd ais_bench/datasets
   git lfs install
   git clone https://huggingface.co/datasets/livecodebench/code_generation_lite
   

6. AIME 2024 dataset.

   cd ais_bench/datasets
   mkdir aime/
   cd aime/
   wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/aime.zip
   unzip aime.zip
   rm aime.zip
   

7. GSM8K dataset.

   cd ais_bench/datasets
   wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gsm8k.zip
   unzip gsm8k.zip
   rm gsm8k.zip
   

Configuration

Update the file benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py. There are several arguments that you should update according to your environment.

• attr: Identifier for the inference backend type, fixed as service (serving-based inference) or local (local model).
• type: Used to select different backend API types.
• abbr: Unique identifier for a local task, used to distinguish between multiple tasks.
• path: Update to your model weight path.
• model: Update to your model name in vLLM.
• host_ip and host_port: Update to your vLLM server ip and port.
• max_out_len: Note max_out_len + LLM input length should be less than max_model_len(config in your vllm server), 32768 will be suitable for most datasets.
• batch_size: Update according to your dataset.
• temperature：设置推理参数。

from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr='vllm-api-general-chat',
        path="xxxx",
        model="xxxx",
        request_rate = 0,
        retry = 2,
        host_ip = "localhost",
        host_port = 8000,
        max_out_len = xxx,
        batch_size = xxx,
        trust_remote_code=False,
        generation_kwargs = dict(
            temperature = 0.6,
            top_k = 10,
            top_p = 0.95,
            seed = None,
            repetition_penalty = 1.03,
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    )
]

执行准确率评估

运行以下代码执行不同的准确率评估。

# run C-Eval dataset
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds

# run MMLU dataset
ais_bench --models vllm_api_general_chat --datasets mmlu_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds

# run GPQA dataset
ais_bench --models vllm_api_general_chat --datasets gpqa_gen_0_shot_str.py --mode all --dump-eval-details --merge-ds

# run MATH-500 dataset
ais_bench --models vllm_api_general_chat --datasets math500_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds

# run LiveCodeBench dataset
ais_bench --models vllm_api_general_chat --datasets livecodebench_code_generate_lite_gen_0_shot_chat.py --mode all --dump-eval-details --merge-ds

# run AIME 2024 dataset
ais_bench --models vllm_api_general_chat --datasets aime2024_gen_0_shot_chat_prompt.py --mode all --dump-eval-details --merge-ds

# run GSM8K dataset
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds

每个数据集执行完毕后，您可以从保存的文件（如 outputs/default/20250628_151326）中获取结果，示例如下：

20250628_151326/
├── configs # Combined configuration file for model tasks, dataset tasks, and result presentation tasks
│   └── 20250628_151326_29317.py
├── logs # Execution logs; if --debug is added to the command, no intermediate logs are saved to disk (all are printed directly to the screen)
│   ├── eval
│   │   └── vllm-api-general-chat
│   │       └── demo_gsm8k.out # Logs of the accuracy evaluation process based on inference results in the predictions/ folder
│   └── infer
│       └── vllm-api-general-chat
│           └── demo_gsm8k.out # Logs of the inference process
├── predictions
│   └── vllm-api-general-chat
│       └── demo_gsm8k.json # Inference results (all outputs returned by the inference service)
├── results
│   └── vllm-api-general-chat
│       └── demo_gsm8k.json # Raw scores calculated from the accuracy evaluation
└── summary
    ├── summary_20250628_151326.csv # Final accuracy scores (in table format)
    ├── summary_20250628_151326.md # Final accuracy scores (in Markdown format)
    └── summary_20250628_151326.txt # Final accuracy scores (in text format)

执行性能评估

纯文本基准测试：

# run C-Eval dataset
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt.py --summarizer default_perf --mode perf

# run MMLU dataset
ais_bench --models vllm_api_general_chat --datasets mmlu_gen_0_shot_cot_chat_prompt.py --summarizer default_perf --mode perf

# run GPQA dataset
ais_bench --models vllm_api_general_chat --datasets gpqa_gen_0_shot_str.py --summarizer default_perf --mode perf

# run MATH-500 dataset
ais_bench --models vllm_api_general_chat --datasets math500_gen_0_shot_cot_chat_prompt.py --summarizer default_perf --mode perf

# run LiveCodeBench dataset
ais_bench --models vllm_api_general_chat --datasets livecodebench_code_generate_lite_gen_0_shot_chat.py --summarizer default_perf --mode perf

# run AIME 2024 dataset
ais_bench --models vllm_api_general_chat --datasets aime2024_gen_0_shot_chat_prompt.py --summarizer default_perf --mode perf

# run GSM8K dataset
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_str_perf.py --summarizer default_perf --mode perf

多模态基准测试（文本 + 图像）：

# run textvqa dataset
ais_bench --models vllm_api_stream_chat --datasets textvqa_gen_base64 --summarizer default_perf --mode perf

执行完成后，您可以从保存的文件中获取结果，示例如下：

20251031_070226/
|-- configs # Combined configuration file for model tasks, dataset tasks, and result presentation tasks
|   `-- 20251031_070226_122485.py
|-- logs
|   `-- performances
|       `-- vllm-api-general-chat
|           `-- cevaldataset.out # Logs of the performance evaluation process
`-- performances
    `-- vllm-api-general-chat
        |-- cevaldataset.csv # Final performance results (in table format)
        |-- cevaldataset.json # Final performance results (in json format)
        |-- cevaldataset_details.h5 # Final performance results in details
        |-- cevaldataset_details.json # Final performance results in details
        |-- cevaldataset_plot.html # Final performance results (in html format)
        `-- cevaldataset_rps_distribution_plot_with_actual_rps.html # Final performance results (in html format)

3.故障排除

无效图像路径错误

如果您按照 AISBench 文档下载 TextVQA 数据集：

cd ais_bench/datasets
git lfs install
git clone https://huggingface.co/datasets/maoxx241/textvqa_subset
mv textvqa_subset/ textvqa/
mkdir textvqa/textvqa_json/
mv textvqa/*.json textvqa/textvqa_json/
mv textvqa/*.jsonl textvqa/textvqa_json/

可能会遇到以下错误：

AISBench - ERROR - /vllm-workspace/benchmark/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: HTTP status 400. Server response: {"error":{"message":"1 validation error for ChatCompletionContentPartImageParam\nimage_url\n  Input should be a valid dictionary [type=dict_type, input_value='data/textvqa/train_images/b2ae0f96dfbea5d8.jpg', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.12/v/dict_type None","type":"BadRequestError","param":null,"code":400}}

您需要手动将数据集图像路径替换为绝对路径，将 /path/to/benchmark/ais_bench/datasets/textvqa/train_images/ 改为实际存储图像的绝对目录：

cd ais_bench/datasets/textvqa/textvqa_json
sed -i 's#data/textvqa/train_images/#/path/to/benchmark/ais_bench/datasets/textvqa/train_images/#g' textvqa_val.json