PaddleOCR-VL

1.简介

PaddleOCR-VL 是一个针对文档解析的 SOTA 级且资源高效的模型。其核心组件是 PaddleOCR-VL-0.9B，这是一个紧凑而强大的视觉语言模型（VLM），它将 NaViT 风格的动态分辨率视觉编码器与 ERNIE-4.5-0.3B 语言模型集成，以实现精确的元素识别。

本文档提供了模型的完整部署和验证的详细工作流程，包括支持的特性、环境准备、单节点部署和功能验证。旨在帮助用户快速完成模型的部署和验证。

本文档基于 vLLM-Ascend v0.21.0rc1 进行验证和编写。当前模型（PaddleOCR-VL）在此版本中受支持。建议使用此版本或其他更新的官方版本进行部署。

2.支持的特性

请参阅支持的特性矩阵获取模型支持的特性列表。

请参阅特性指南获取特性的配置方法。

3.前提条件

3.1.模型权重

• PaddleOCR-VL-0.9B：PaddleOCR-VL-0.9B

建议将模型权重下载到缓存目录，并设置 VLLM_USE_MODELSCOPE=True 以自动加载模型。如果已将权重下载到本地目录，请相应更新部署脚本中的 MODEL_PATH 变量。

4.安装

4.1.Docker 镜像安装

您可以使用官方 Docker 镜像直接运行 PaddleOCR-VL。

根据您的机器类型选择镜像，并在节点上启动 Docker 镜像，请参考使用 Docker 安装。

A2 系列Atlas 300 inference products

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-310p
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

After a successful docker run, you can verify the running container service by executing the docker ps command.

4.2 Source Code Installation

If you don't want to use the docker image as above, you can also build all from source:

• Install vllm-ascend from source, refer to installation.

Note

If you are using Atlas 300 inference products, you may need to uninstall triton-ascend to avoid dependency conflicts.

5 Online Service Deployment

5.1 Single-Node Online Deployment

PaddleOCR-VL supports single-node single-card deployment on the A2 series and Atlas 300 inference products platform. Single-node deployment completes both Prefill and Decode within the same node.

Follow these steps to start the inference service:

1. Prepare model weights: Ensure the model weights are accessible. With VLLM_USE_MODELSCOPE=True, the model will be loaded automatically from ModelScope.
2. Set the MODEL_PATH environment variable to point to your model directory.
3. Create and execute the deployment script (save as deploy.sh).

Startup Command:

A2 seriesAtlas 300 inference products

#!/bin/sh
export VLLM_USE_MODELSCOPE=True
export MODEL_PATH="PaddlePaddle/PaddleOCR-VL"
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

vllm serve ${MODEL_PATH} \
          --max-num-batched-tokens 16384 \
          --served-model-name PaddleOCR-VL-0.9B \
          --trust-remote-code \
          --no-enable-prefix-caching \
          --mm-processor-cache-gb 0 \
          --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
          --additional_config '{"enable_cpu_binding":true}' \
          --port 8000

#!/bin/sh
export VLLM_USE_MODELSCOPE=True
export MODEL_PATH="PaddlePaddle/PaddleOCR-VL"
export TASK_QUEUE_ENABLE=1
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

vllm serve ${MODEL_PATH} \
          --max_model_len 16384 \
          --served-model-name PaddleOCR-VL-0.9B \
          --trust-remote-code \
          --no-enable-prefix-caching \
          --mm-processor-cache-gb 0 \
          --dtype float16 \
          --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
          --additional_config '{"ascend_compilation_config": {"fuse_norm_quant": false}, "enable_cpu_binding":true}' \
          --port 8000

Note

On Atlas 300 inference products:

• Only float16 dtype is supported.
• The --max_model_len option is added to prevent errors when generating the attention operator mask.
• Graph compilation (--compilation-config) requires CANN version >= 9.0.0. If your CANN version is lower, please revert to eager mode by replacing the --compilation-config argument with --enforce-eager.
• The --additional_config option in fuse_norm_quant is disabled (false) because it is not supported by the graph compilation on this hardware. Keep this setting unchanged.

Key Parameter Descriptions:

• --max-num-batched-tokens specifies the maximum number of tokens batched in a single forward pass. Adjust this parameter for throughput optimization.
• --max_model_len specifies the maximum context length — that is, the sum of input and output tokens for a single request.
• --no-enable-prefix-caching indicates that prefix caching is disabled. To enable it, remove this option.
• --mm-processor-cache-gb sets the size of the multimodal processor cache (in GB). A value of 0 disables caching.
• --数据类型 float16 specifies the model dtype. On Atlas 300 inference products, only float16 is supported.
• --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' enables full decode graph compilation for improved performance. On Atlas 300 inference products, --additional_config in graph compilation is disabled by default in fuse_norm_quant.
• --additional_config '{"enable_cpu_binding":true}' enables CPU binding to improve performance.

Common Issues Tip: If you encounter startup issues, please refer to the Public FAQ for troubleshooting.

5.2 Multi-Node PD Separation Deployment

Not supported yet.

5.3 Special Deployment Modes

5.3.1 Offline Inference with vLLM and PP-DocLayoutV2

In the above example, we demonstrated how to use vLLM to infer the PaddleOCR-VL-0.9B model. Typically, we also need to integrate the PP-DocLayoutV2 model to fully unleash the capabilities of the PaddleOCR-VL model, making it more consistent with the examples provided by the official PaddlePaddle documentation.

Note

Use separate virtual environments for VLLM and PP-DocLayoutV2 to prevent dependency conflicts.

A2 seriesAtlas 300 推理产品

The A2 series device supports inference using the PaddlePaddle framework.

1. Pull the PaddlePaddle-compatible CANN image

   docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/device/paddle-npu:cann800-ubuntu20-npu-910b-base-aarch64-gcc84
   

   Start the container using the following command:

   docker run -it --name paddle-npu-dev -v $(pwd):/work \
       --privileged --network=host --shm-size=128G -w=/work \
       -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
       -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
       -v /usr/local/dcmi:/usr/local/dcmi \
       -e ASCEND_RT_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" \
       ccr-2vdh3abv-pub.cnc.bj.baidubce.com/device/paddle-npu:cann800-ubuntu20-npu-910b-base-$(uname -m)-gcc84 /bin/bash
   

2. Install PaddlePaddle and PaddleOCR

   python -m pip install paddlepaddle==3.2.0
   wget https://paddle-whl.bj.bcebos.com/stable/npu/paddle-custom-npu/paddle_custom_npu-3.2.0-cp310-cp310-linux_aarch64.whl
   pip  install  paddle_custom_npu-3.2.0-cp310-cp310-linux_aarch64.whl
   python -m pip install -U "paddleocr[doc-parser]"
   pip install safetensors
   

   Note

   The OpenCV component may be missing:
   
   ```bash
   apt-get update
   apt-get install -y libgl1 libglib2.0-0
   ```
   
   CANN-8.0.0 does not support some versions of NumPy and OpenCV. It is recommended to install the specified versions.
   
   ```bash
   python -m pip install numpy==1.26.4
   python -m pip install opencv-python==3.4.18.65
   ```
   

Atlas 300 推理产品仅支持 OM 模型推理。有关过程的详细信息，请参阅 ModelZoo 中提供的指南。

5.3.2 使用 vLLM 作为后端，结合 PP-DocLayoutV2 进行离线推理

from paddleocr import PaddleOCRVL

doclayout_model_path = "/path/to/your/PP-DocLayoutV2/"

pipeline = PaddleOCRVL(vl_rec_backend="vllm-server", 
                       vl_rec_server_url="http://localhost:8000/v1", 
                       layout_detection_model_name="PP-DocLayoutV2",  
                       layout_detection_model_dir=doclayout_model_path,
                       device="npu")

output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")

for i, res in enumerate(output):
    res.save_to_json(save_path=f"output_{i}.json")
    res.save_to_markdown(save_path=f"output_{i}.md")

6 功能验证

如果服务启动成功，您将看到如下所示的信息：

INFO:     Started server process [87471]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

服务器启动后，您可以使用 OpenAI API 客户端进行查询。

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

# Task-specific base prompts
TASKS = {
    "ocr": "OCR:",
    "table": "Table Recognition:",
    "formula": "Formula Recognition:",
    "chart": "Chart Recognition:",
}

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": TASKS["ocr"]
            }
        ]
    }
]

response = client.chat.completions.create(
    model="PaddleOCR-VL-0.9B",
    messages=messages,
    temperature=0.0,
)
print(f"Generated text: {response.choices[0].message.content}")

预期结果：

如果查询服务器成功，您将看到如下所示的信息（客户端）：

Generated text: CINNAMON SUGAR
1 x 17,000
17,000
SUB TOTAL
17,000
GRAND TOTAL
17,000
CASH IDR
20,000
CHANGE DUE
3,000

7 精度评估

有关 PaddleOCR-VL 的精度评估，请参阅官方 ModelZoo 中的评估流程和结果。

8 性能评估

有关 PaddleOCR-VL 的性能评估，请参阅官方 ModelZoo 中的基准测试方法和结果。

9 性能调优

9.1 推荐配置

注意：以下配置在特定测试环境中验证，仅供参考。最佳配置取决于最大输入/输出长度、精度要求和实际硬件规格等因素。建议根据实际情况参考第 9.2 节进行调优。

PaddleOCR-VL 是一个轻量级模型，在单张 NPU 上运行。关键调优参数在不同硬件平台上有所不同。

表 1：场景概览

场景	硬件	*NPU 总数	权重版本	关键注意事项
高吞吐	A2 系列	1	PaddleOCR-VL-0.9B	-
高吞吐	Atlas 300 推理产品	1	PaddleOCR-VL-0.9B	图编译需要 CANN >= 9.0.0

*Total NPUs 表示所有节点上使用的 NPU 总数。

表 2：详细节点配置

场景	配置	NPU 数量	TP	DP	最大模型长度	最大批处理 Token 数	图编译	数据类型
高吞吐	A2 系列 / 单机	1	—	—	—	16384	FULL_DECODE_ONLY	bfloat16（默认）
高吞吐	Atlas 300 推理产品 / 单机	1	—	—	16384	—	FULL_DECODE_ONLY；否则使用 enforce-eager	float16

有关完整的启动命令和参数说明，请参阅第 5.1 节中的部署示例。

9.2 调优指南

9.2.1 通用调优参考

有关性能调优，请参阅公共性能调优文档了解通用调优方法，包括操作系统优化（jemalloc、tcmalloc）、torch_npu 优化（内存和调度）以及 CANN 优化。

请参阅特性指南获取详细的特性描述。

10 常见问题

For common environment, installation, and general parameter issues, please refer to the Public FAQ; this chapter only covers model-specific issues.

• 问：Atlas 300 推理产品的部署要求是什么？

答：在 Atlas 300 推理产品上，仅支持 float16 数据类型。图编译（--compilation-config）需要 CANN 版本 >= 9.0.0；如果您的 CANN 版本较低，请改用 --enforce-eager。

• 问：在 Atlas 300 推理产品上安装时遇到依赖冲突怎么办？

答：您可能需要卸载 triton-ascend 以避免依赖冲突。详见第 4.2 节。