Kimi-K3

Kimi-K3#

1 Introduction#

Kimi K3 is a native multimodal Mixture-of-Experts (MoE) model. Its language backbone combines Kimi Delta Attention (KDA) with periodic Gated Multi-head Latent Attention (MLA), and uses Stable LatentMoE for expert computation. The model also integrates a MoonViT vision encoder and supports text, image understanding, reasoning, and tool calling.

This document will show the main verification steps of the model, including supported features, feature configuration, environment preparation, multi-node deployment on Atlas 800 A3 and Atlas 800 A2, functional verification, and accuracy and performance evaluation.

This document is validated and written based on vLLM-Ascend 0.23.0. The current model (Kimi-K3) is first supported in this version.

2 Supported Features#

Refer to supported features to get the model’s supported feature matrix.

Refer to feature guide to get the feature’s configuration.

3 Prerequisites#

3.1 Model Weight#

Download the Eco-Tech/Kimi-K3-w4a8 ModelSlim W4A8 quantized weight from ModelScope. This guide includes the following validated deployment configurations:

Platform	Deployment	Topology
4 × Atlas 800 A3 (64G × 16)	Mixed Prefill/Decode deployment	DP4/TP16/EP64
16 × Atlas 800 A3 (64G × 16)	Eight Prefill nodes and eight Decode nodes	DP8/TP16 on each side
8 × Atlas 800 A2 (64G × 8)	Mixed Prefill/Decode deployment	DP8/TP8/EP64

The checkpoint directory must contain the model configuration, tokenizer, image processor, and model weight files required by the published Kimi K3 package.

It is recommended to download the model weight to the shared directory of multiple nodes, such as /root/.cache/.

3.2 Verify Multi-node Communication (Optional)#

If you want to deploy multi-node environment, you need to verify multi-node communication according to verify multi-node communication environment.

4 Installation#

4.1 Docker Image Installation#

4.1.1 Atlas 800 A3#

Kimi K3 is validated on Atlas 800 A3 (64G × 16). Select the image that matches the host operating system and start it on each node, referring to using docker.

Host operating system	Image
Ubuntu	`quay.io/ascend/vllm-ascend:kimi-k3-a3`
openEuler	`quay.io/ascend/vllm-ascend:kimi-k3-a3-openeuler`

Run the following command on each node:

# Ubuntu:
export IMAGE=quay.io/ascend/vllm-ascend:kimi-k3-a3
# openEuler:
# export IMAGE=quay.io/ascend/vllm-ascend:kimi-k3-a3-openeuler
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

After a successful docker run, you can verify the running container service by executing the docker ps command.

4.1.2 Atlas 800 A2#

Kimi K3 is validated on Atlas 800 A2 (64G × 8). Select the image that matches the host operating system and start it on each node, referring to using docker.

Host operating system	Image
Ubuntu	`quay.io/ascend/vllm-ascend:kimi-k3`
openEuler	`quay.io/ascend/vllm-ascend:kimi-k3-openeuler`

Run the following command on each node:

# Ubuntu:
export IMAGE=quay.io/ascend/vllm-ascend:kimi-k3
# openEuler:
# export IMAGE=quay.io/ascend/vllm-ascend:kimi-k3-openeuler
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

After a successful docker run, you can verify the running container service by executing the docker ps command.

4.2 Source Code Installation#

If you don’t want to use the docker image as above, you can also build all from source:

Install vllm-ascend from source, refer to installation.

If you want to deploy multi-node environment, you need to set up environment on each node.

Kimi K3 configuration, multimodal processing, reasoning parsing, and tool parsing are registered by vLLM-Ascend. Use a vLLM and vLLM-Ascend source revision that matches the validated version in this document.

5 Online Service Deployment#

5.1 Atlas 800 A3 Deployments#

5.1.1 Four-Node Mixed Deployment#

The validated mixed deployment uses four Atlas 800 A3 (64G × 16) nodes. vLLM data parallelism spans the four nodes, each node runs one DP rank, and tensor parallelism uses all 16 NPUs in the node. The resulting topology is DP4/TP16/EP64.

Before starting the service:

Replace the model path, local IP address, network interface, service port, and DP RPC port with values from the target environment.
NIC_NAME must be the interface that owns LOCAL_IP.
Start Node 0 first. The NODE0_IP configured on Nodes 1 through 3 must equal LOCAL_IP on Node 0.
Assign --data-parallel-start-rank values 1, 2, and 3 to Nodes 1, 2, and 3 respectively.

Node 0

# Values that must be adapted to the target environment.
export MODEL_PATH=<KIMI_K3_MODEL_PATH>
export LOCAL_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<NODE0_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>

export HCCL_IF_IP=$LOCAL_IP
export GLOO_SOCKET_IFNAME=$NIC_NAME
export TP_SOCKET_IFNAME=$NIC_NAME
export HCCL_SOCKET_IFNAME=$NIC_NAME
export VLLM_ENGINE_READY_TIMEOUT_S=7200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export TASK_QUEUE_ENABLE=1
export HCCL_BUFFSIZE=800
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

vllm serve $MODEL_PATH \
    --served-model-name kimi-k3 \
    --port $PORT \
    --allowed-local-media-path / \
    --trust-remote-code \
    --tensor-parallel-size 16 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-address $LOCAL_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --enable-prefix-caching \
    --enable-expert-parallel \
    --max-num-seqs 16 \
    --max-model-len 131072 \
    --max-num-batched-tokens 24576 \
    --gpu-memory-utilization 0.9 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --mm-processor-cache-gb 0 \
    --additional-config '{"enable_cpu_binding":true, "enable_flashcomm1":true}' \
    --mm-encoder-tp-mode data \
    --limit-mm-per-prompt '{"vision_chunk": 2}' \
    --enable-auto-tool-choice \
    --reasoning-parser kimi_k3 \
    --tool-call-parser kimi_k3

Nodes 1-3

Run this command on every worker node. Set LOCAL_IP and NIC_NAME to the current node and set DP_START_RANK to 1, 2, or 3.

# Values that must be adapted to the target environment.
export MODEL_PATH=<KIMI_K3_MODEL_PATH>
export LOCAL_IP=<WORKER_LOCAL_IP>
export NODE0_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<WORKER_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>
export DP_START_RANK=<1_OR_2_OR_3>

export HCCL_IF_IP=$LOCAL_IP
export GLOO_SOCKET_IFNAME=$NIC_NAME
export TP_SOCKET_IFNAME=$NIC_NAME
export HCCL_SOCKET_IFNAME=$NIC_NAME
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export TASK_QUEUE_ENABLE=1
export HCCL_BUFFSIZE=800
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

vllm serve $MODEL_PATH \
    --headless \
    --served-model-name kimi-k3 \
    --port $PORT \
    --allowed-local-media-path / \
    --trust-remote-code \
    --tensor-parallel-size 16 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank $DP_START_RANK \
    --data-parallel-address $NODE0_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --enable-prefix-caching \
    --enable-expert-parallel \
    --max-num-seqs 16 \
    --max-model-len 131072 \
    --max-num-batched-tokens 24576 \
    --gpu-memory-utilization 0.9 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --mm-processor-cache-gb 0 \
    --additional-config '{"enable_cpu_binding":true, "enable_flashcomm1":true}' \
    --mm-encoder-tp-mode data \
    --limit-mm-per-prompt '{"vision_chunk": 2}' \
    --enable-auto-tool-choice \
    --reasoning-parser kimi_k3 \
    --tool-call-parser kimi_k3

The following values differ between the master and worker nodes:

Setting	Node 0	Nodes 1-3	Description
`LOCAL_IP`	Node 0 IP	Current worker IP	Each node uses its own IP address.
`NODE0_IP`	Not required	Node 0 IP	Workers use this address to join the DP group.
`VLLM_ENGINE_READY_TIMEOUT_S`	`7200`	Not set	Only the master waits for all engines to become ready.
`--headless`	Omitted	Enabled	Workers do not expose the API endpoint.
`--data-parallel-address`	`$LOCAL_IP`	`$NODE0_IP`	Always resolves to Node 0.
`--data-parallel-start-rank`	`0` by default	`1`, `2`, or `3`	Every node must own a unique DP rank.

Key deployment parameters:

Parameter	Description
`--tensor-parallel-size 16`	Uses all 16 NPUs in one A3 node for tensor parallelism.
`--data-parallel-size 4`	Creates four global DP ranks across four nodes.
`--data-parallel-size-local 1`	Runs one DP rank on the current node.
`--data-parallel-start-rank`	Selects the global starting DP rank for a worker node.
`--data-parallel-rpc-port`	Must be identical and reachable on every node.
`--enable-expert-parallel`	Enables expert parallelism for the MoE layers.
`--max-model-len 131072`	Sets the maximum combined input and output length.
`--max-num-seqs 16`	Sets the maximum active sequences for each DP group.
`--max-num-batched-tokens 24576`	Controls the scheduler token budget.
`--enable-prefix-caching`	Enables automatic prefix caching.
`--compilation-config`	Uses `FULL_DECODE_ONLY` ACL Graph replay.
`--additional-config`	Enables Ascend CPU binding and FlashComm1.
`HCCL_IF_IP` and socket interface variables	Bind HCCL, Gloo, and TP communication to the selected interface.

Note

Serving a 1M-token context requires at least eight Atlas 800 A3 (64G × 16) nodes. Change the following parameters on every node:

Parameter	Four-node default	Eight-node (1M context)
`--data-parallel-size`	`4`	`8`
`--max-model-len`	`131072`	`1048576`
`--max-num-batched-tokens`	`24576`	`8192`

Run the worker command on Nodes 1 through 7 and assign each node a unique --data-parallel-start-rank from 1 through 7.

If a worker exits immediately, confirm that Node 0 is already running, --data-parallel-address resolves to Node 0, and every worker uses a unique --data-parallel-start-rank.

Verify the service through Node 0:

curl http://<NODE0_LOCAL_IP>:<SERVICE_PORT>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "kimi-k3",
        "messages": [{
            "role": "user",
            "content": [{
                "type": "text",
                "text": "The future of AI is"
            }]
        }],
        "max_tokens": 1024,
        "temperature": 1.0,
        "top_p": 0.95
    }'

The service should return HTTP 200 and a choices field containing generated text.

5.1.2 Sixteen-Node PD Separation Deployment#

The validated PD separation topology uses 16 Atlas 800 A3 (64G × 16) nodes: eight Prefill nodes and eight Decode nodes. Both sides use DP8/TP16.

Refer to PD Disaggregation with Mooncake for the general service workflow.

5.1.2.2 Create the engine templates#

Prefill

KV_PORT=36000

unset ftp_proxy FTP_PROXY
unset https_proxy HTTPS_PROXY
unset http_proxy HTTP_PROXY

export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=120

nic_name=<PREFILL_NIC_NAME>
local_ip=<PREFILL_LOCAL_IP>

export HCCL_IF_IP=${local_ip}
export GLOO_SOCKET_IFNAME=${nic_name}
export TP_SOCKET_IFNAME=${nic_name}
export HCCL_SOCKET_IFNAME=${nic_name}
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_ENABLE_MLAPO=1
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export VLLM_USE_V1=1
export ASCEND_RT_VISIBLE_DEVICES=$1
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

export PYTHONHASHSEED=0
export ACL_OP_INIT_MODE=1

vllm serve <KIMI_K3_MODEL_PATH> \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name kimi-k3 \
    --max-model-len 133120 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 16 \
    --enforce-eager \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --mm-encoder-tp-mode data \
    --skip-mm-profiling \
    --safetensors_load_strategy prefetch \
    --mamba-cache-mode align \
    --enable-prefix-caching \
    --additional-config '{"recompute_scheduler_enable":false}' \
    --limit-mm-per-prompt '{"vision_chunk": 2}' \
    --kv-transfer-config \
    '{
      "kv_connector": "MooncakeConnectorV1",
      "kv_role": "kv_producer",
      "kv_port": "'"$KV_PORT"'",
      "kv_connector_extra_config": {
        "prefill": {
            "dp_size": 8,
            "tp_size": '"$7"'
        },
        "decode": {
            "dp_size": 8,
            "tp_size": '"$7"'
        }
      }
    }'

Decode

KV_PORT=36200

unset ftp_proxy FTP_PROXY
unset https_proxy HTTPS_PROXY
unset http_proxy HTTP_PROXY

export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=120

nic_name=<DECODE_NIC_NAME>
local_ip=<DECODE_LOCAL_IP>

export HCCL_IF_IP=${local_ip}
export GLOO_SOCKET_IFNAME=${nic_name}
export TP_SOCKET_IFNAME=${nic_name}
export HCCL_SOCKET_IFNAME=${nic_name}
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_ENABLE_MLAPO=1
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_USE_V1=1
export ASCEND_RT_VISIBLE_DEVICES=$1
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

vllm serve <KIMI_K3_MODEL_PATH> \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name kimi-k3 \
    --max-model-len 133120 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 16 \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --mm-encoder-tp-mode data \
    --skip-mm-profiling \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --safetensors_load_strategy prefetch \
    --mamba-cache-mode align \
    --enable-prefix-caching \
    --additional-config '{"recompute_scheduler_enable":false}' \
    --limit-mm-per-prompt '{"vision_chunk":2}' \
    --kv-transfer-config \
    '{
      "kv_connector": "MooncakeConnectorV1",
      "kv_role": "kv_consumer",
      "kv_port": "'"$KV_PORT"'",
      "kv_connector_extra_config": {
        "prefill": {
            "dp_size": 8,
            "tp_size": '"$7"'
        },
        "decode": {
            "dp_size": 8,
            "tp_size": '"$7"'
        }
      }
    }'

5.1.2.3 Start the engines#

Deploy launch_online_dp.py and the corresponding engine template on every node. The following example starts one local DP rank in a DP8/TP16 group:

python launch_online_dp.py \
    --dp-size 8 \
    --tp-size 16 \
    --dp-size-local 1 \
    --dp-rank-start <LOCAL_DP_RANK> \
    --dp-address <PD_MASTER_IP> \
    --dp-rpc-port <DP_RPC_PORT> \
    --vllm-start-port <VLLM_START_PORT>

Use ranks 0 through 7 for each eight-node side. Configure independent master addresses, RPC ports, and vLLM port ranges for the Prefill and Decode groups.

After the engines start, configure and start the load-balancing proxy as described in PD Disaggregation with Mooncake.

Key PD settings:

Setting	Value	Description
Topology	8P8D	Eight Prefill and eight Decode nodes.
`--dp-size`	`8`	Eight DP ranks on each side.
`--tp-size`	`16`	Uses all 16 NPUs in a node.
`--dp-size-local`	`1`	One DP rank per node.
`KV_PORT`	`36000` for P, `36200` for D	Separates producer and consumer KV traffic.
`recompute_scheduler_enable`	`false`	Matches the validated Prefill and Decode configuration.

5.2 Atlas 800 A2 Deployment#

5.2.1 Eight-Node Mixed Deployment#

The validated Atlas 800 A2 deployment uses eight nodes with eight NPUs per node. Each node runs one DP rank and uses all eight local NPUs for tensor parallelism. Every DP rank handles both Prefill and Decode, resulting in a DP8/TP8/EP64 topology. Node 0 runs the API server and DP rank 0, while Nodes 1 through 7 run headless DP workers. This baseline serves the language model only.

Before starting the service:

Replace the model path, local IP address, network interface, service port, and DP RPC port with values from the target environment.
NIC_NAME must be the interface that owns LOCAL_IP.
Start Node 0 first. The NODE0_IP configured on Nodes 1 through 7 must equal LOCAL_IP on Node 0.
Assign a unique DP_START_RANK from 1 through 7 to each worker node.
Ensure proxy bypass settings include all API and communication IP addresses used by the eight nodes.

Node 0

# Values that must be adapted to the target environment.
export MODEL_PATH=<KIMI_K3_MODEL_PATH>
export LOCAL_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<NODE0_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /vllm-workspace/vllm-ascend/vllm_ascend/_cann_ops_custom/vendors/custom_transformer/bin/set_env.bash

export PYTHONPATH=/vllm-workspace/vllm-ascend:${PYTHONPATH:-}
export VLLM_HOST_IP=$LOCAL_IP
export HCCL_IF_IP=$LOCAL_IP
export GLOO_SOCKET_IFNAME=$NIC_NAME
export TP_SOCKET_IFNAME=$NIC_NAME
export HCCL_SOCKET_IFNAME=$NIC_NAME
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_EXEC_TIMEOUT=1800
export HCCL_BUFFSIZE=256
export HCCL_INTRA_ROCE_ENABLE=1
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_LOGGING_LEVEL=INFO
export TASK_QUEUE_ENABLE=1
export TIKTOKEN_CACHE_DIR=/root/.cache/tiktoken-k3
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port $PORT \
    --served-model-name kimi-k3 \
    --trust-remote-code \
    --language-model-only \
    --mm-encoder-tp-mode data \
    --skip-mm-profiling \
    --limit-mm-per-prompt '{"vision_chunk":2}' \
    --data-parallel-size 8 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank 0 \
    --data-parallel-address $LOCAL_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --dtype bfloat16 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8]}' \
    --tokenizer-mode kimi_k3 \
    --enable-auto-tool-choice \
    --reasoning-parser kimi_k3 \
    --tool-call-parser kimi_k3 \
    --additional-config '{"enable_flashcomm1":false,"ascend_compilation_config":{"enable_npugraph_ex":true,"enable_static_kernel":false},"enable_cpu_binding":true}'

Nodes 1-7

Run this command on every worker node. Set LOCAL_IP and NIC_NAME to the current node and set DP_START_RANK to a unique value from 1 through 7.

# Values that must be adapted to the target environment.
export MODEL_PATH=<KIMI_K3_MODEL_PATH>
export LOCAL_IP=<WORKER_LOCAL_IP>
export NODE0_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<WORKER_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>
export DP_START_RANK=<1_TO_7>

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /vllm-workspace/vllm-ascend/vllm_ascend/_cann_ops_custom/vendors/custom_transformer/bin/set_env.bash

export PYTHONPATH=/vllm-workspace/vllm-ascend:${PYTHONPATH:-}
export VLLM_HOST_IP=$LOCAL_IP
export HCCL_IF_IP=$LOCAL_IP
export GLOO_SOCKET_IFNAME=$NIC_NAME
export TP_SOCKET_IFNAME=$NIC_NAME
export HCCL_SOCKET_IFNAME=$NIC_NAME
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_EXEC_TIMEOUT=1800
export HCCL_BUFFSIZE=256
export HCCL_INTRA_ROCE_ENABLE=1
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_LOGGING_LEVEL=INFO
export TASK_QUEUE_ENABLE=1
export TIKTOKEN_CACHE_DIR=/root/.cache/tiktoken-k3
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

vllm serve $MODEL_PATH \
    --headless \
    --host 0.0.0.0 \
    --port $PORT \
    --served-model-name kimi-k3 \
    --trust-remote-code \
    --language-model-only \
    --mm-encoder-tp-mode data \
    --skip-mm-profiling \
    --limit-mm-per-prompt '{"vision_chunk":2}' \
    --data-parallel-size 8 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank $DP_START_RANK \
    --data-parallel-address $NODE0_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --dtype bfloat16 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8]}' \
    --tokenizer-mode kimi_k3 \
    --enable-auto-tool-choice \
    --reasoning-parser kimi_k3 \
    --tool-call-parser kimi_k3 \
    --additional-config '{"enable_flashcomm1":false,"ascend_compilation_config":{"enable_npugraph_ex":true,"enable_static_kernel":false},"enable_cpu_binding":true}'

The following values differ between the master and worker nodes:

Setting	Node 0	Nodes 1-7	Description
`LOCAL_IP`	Node 0 IP	Current worker IP	Each node uses its own communication IP address.
`NODE0_IP`	Not required	Node 0 IP	Workers use this address to join the DP group.
`--headless`	Omitted	Enabled	Workers do not expose an API endpoint.
`--data-parallel-address`	`$LOCAL_IP`	`$NODE0_IP`	Always resolves to Node 0.
`--data-parallel-start-rank`	`0`	Unique value from `1` through `7`	Every node owns one global DP rank.

Key A2 deployment parameters:

Parameter	Description
`--tensor-parallel-size 8`	Uses all eight NPUs in one A2 node for tensor parallelism.
`--data-parallel-size 8`	Creates eight global DP ranks across eight nodes.
`--data-parallel-size-local 1`	Runs one DP rank on the current node.
`--language-model-only`	Disables the multimodal encoder for this validated A2 baseline.
`--max-model-len 262144`	Sets a 256K combined input and output context limit.
`--compilation-config`	Uses `FULL_DECODE_ONLY` graph replay with capture sizes `1`, `2`, `4`, and `8`.
`--additional-config`	Enables NPU graph execution and CPU binding while keeping FlashComm1 disabled.

Do not set HCCL_OP_EXPANSION_MODE=AIV for this baseline. Start Node 0 first, then start Nodes 1 through 7 as soon as possible. If a worker exits immediately, verify that Node 0 is running, all nodes use the same RPC port, --data-parallel-address resolves to Node 0, and every worker has a unique DP start rank.

6 Functional Verification#

6.1 Atlas 800 A3#

After an A3 mixed or PD service is ready, send a multimodal request to the API endpoint:

curl http://<SERVICE_IP>:<SERVICE_PORT>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "kimi-k3",
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": "<IMAGE_URL_OR_DATA_URL>"}
                },
                {
                    "type": "text",
                    "text": "Describe the image."
                }
            ]
        }],
        "max_tokens": 1024,
        "temperature": 1.0,
        "top_p": 0.95
    }'

The service should return HTTP 200 and a choices field containing the image description. The current implementation supports image inputs but does not support video inputs.

6.2 Atlas 800 A2#

The validated A2 deployment uses --language-model-only. After all eight DP ranks are ready, send a text request to the Node 0 API endpoint:

curl http://<NODE0_LOCAL_IP>:<SERVICE_PORT>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "kimi-k3",
        "messages": [{
            "role": "user",
            "content": "Explain data parallelism in one sentence."
        }],
        "max_tokens": 64
    }'

The service should return HTTP 200 and a choices field containing generated text. Nodes 1 through 7 are headless workers and do not accept HTTP requests directly.

X-data-parallel-rank is an optional HTTP request header that pins a request to a specific DP rank. Without this header, the internal vLLM load balancer on Node 0 selects an available rank. For this DP8 deployment, use an integer from 0 through 7 only when validating one rank, troubleshooting a worker, or testing rank-local prefix-cache behavior:

-H "X-data-parallel-rank: 0" \

Production traffic should normally omit this header so that requests remain balanced across all DP ranks. The request is always sent to the Node 0 API endpoint, even when a worker rank is selected.

7 Accuracy Evaluation#

Here is one accuracy evaluation method for Kimi K3.

Using AISBench#

Refer to Using AISBench for the environment setup and evaluation procedure.
Run AISBench against the Kimi K3 service and collect the generated result files. Keep the model and tokenizer revisions, chat template, sampling settings, dataset, and evaluator revision fixed when comparing results.

8 Performance Evaluation#

Using AISBench#

Refer to Using AISBench for performance evaluation for the environment setup and evaluation procedure.

Using vLLM Benchmark#

Use vllm bench serve to measure the online serving performance of the Kimi K3 service. The following is a minimal example for eight 128k-input, 1k-output requests. Replace the service address, model path, dataset path, and result directory for the target environment.

Refer to vllm benchmark for more details.

export DATA_NUM=8
export CONCURRENCY=8

vllm bench serve \
  --base-url http://<SERVICE_HOST>:<SERVICE_PORT> \
  --endpoint /v1/completions \
  --model kimi-k3 \
  --tokenizer <KIMI_K3_MODEL_PATH> \
  --tokenizer-mode kimi_k3 \
  --trust-remote-code \
  --dataset-name custom \
  --dataset-path <DATASET_PATH> \
  --custom-output-len 1024 \
  --skip-chat-template \
  --num-prompts ${DATA_NUM} \
  --request-rate inf \
  --max-concurrency ${CONCURRENCY} \
  --ignore-eos \
  --temperature 0 \
  --seed 0 \
  --disable-shuffle \
  --save-result \
  --result-dir <RESULT_DIR>

9 Performance Tuning#

Use the validated deployment values above as a baseline. Adjust max-model-len, max-num-seqs, max-num-batched-tokens, and gpu-memory-utilization together for the target workload.

Refer to the performance tuning guide and the feature matrix for additional guidance.

10 FAQ#

For common environment, installation, and general parameter issues, refer to the Public FAQ.

Q: Which multimodal inputs are supported by the current Kimi K3 implementation?

A: The current local processor accepts image inputs. Video inputs are not supported.
Q: Which server options are required for Kimi K3 reasoning and tool calling?

A: Configure --enable-auto-tool-choice, --reasoning-parser kimi_k3, and --tool-call-parser kimi_k3 together.
Q: How should TP size be selected?

A: TP size must divide the checkpoint’s attention-head count. It also affects KDA state layout and expert placement, so validate memory capacity and communication performance together.

Kimi-K3

Contents

Kimi-K3#

1 Introduction#

2 Supported Features#

3 Prerequisites#

3.1 Model Weight#

3.2 Verify Multi-node Communication (Optional)#

4 Installation#

4.1 Docker Image Installation#

4.1.1 Atlas 800 A3#

4.1.2 Atlas 800 A2#

4.2 Source Code Installation#

5 Online Service Deployment#

5.1 Atlas 800 A3 Deployments#

5.1.1 Four-Node Mixed Deployment#

5.1.2 Sixteen-Node PD Separation Deployment#

5.1.2.2 Create the engine templates#

5.1.2.3 Start the engines#

5.2 Atlas 800 A2 Deployment#

5.2.1 Eight-Node Mixed Deployment#

6 Functional Verification#

6.1 Atlas 800 A3#

6.2 Atlas 800 A2#

7 Accuracy Evaluation#

Using AISBench#

8 Performance Evaluation#

Using AISBench#

Using vLLM Benchmark#

9 Performance Tuning#

10 FAQ#