Skip to content

Minitron-8B-Base

Introduction

The released Minitron-8B-Base is a lightweight, efficient large language model developed by NVIDIA. It is designed for general-purpose text generation and reasoning tasks, and can be deployed with vLLM for online serving and evaluation on Ascend NPU hardware through vllm-ascend.

This document describes the main verification steps of the model, including supported features, environment preparation, single-node deployment, functional verification, and accuracy evaluation on the GSM8K benchmark.

Environment Preparation

Model Weight

Minitron-8B-Base(BF16 version): requires 1 Ascend 910B (with 1 x 64G NPUs). Download model weight

It is recommended to place the model weight in a shared cache directory, such as /root/.cache/ or a local model path like /data/vllm-workspace/models/Minitron-8B-Base.

Installation

Minitron-8B-Base can be deployed with vllm-ascend in a compatible runtime environment.

You can use the official docker image for deployment:

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
  --name vllm-ascend \
  --shm-size=1g \
  --device /dev/davinci0 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /root/.cache:/root/.cache \
  -v /data/vllm-workspace/models:/data/vllm-workspace/models \
  -p 8000:8000 \
  -it $IMAGE bash

If you do not want to use the docker image, you can also build from source:

Deployment

Start the online serving service with the following command:

vllm serve "nv-community/Minitron-8B-Base" \
  --served-model-name minitron-8b-base \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --enforce-eager \
  --port 8000

Functional Verification

Once your server is started, you can query the model with a simple prompt:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "minitron-8b-base",
    "prompt": "Question: If a train travels 60 miles in 2 hours, what is its average speed in miles per hour?\nAnswer:",
    "max_tokens": 64,
    "temperature": 1.0
  }'

A valid response indicates that the model is deployed correctly and can generate text outputs.

Accuracy Evaluation

The GSM8K dataset was used to evaluate the reasoning capability of Minitron-8B-Base.

The current evaluation setting is:

  • Dataset: gsm8k
  • Split: test
  • Number of samples: 1000
  • Few-shot setting: 5-shot
  • apply_chat_template: False
  • fewshot_as_multiturn: False

The current evaluation results are:

Category Dataset Metric Result
Accuracy gsm8k / test Total Samples 1000
Accuracy gsm8k / test exact_match,strict-match 0.5436
Accuracy gsm8k / test exact_match,flexible-extract 0.5451

Remarks on Metrics

  • exact_match,strict-match: Only predictions that strictly match the expected final-answer extraction format are counted as correct.
  • exact_match,flexible-extract: Predictions are evaluated with a more flexible answer extraction rule, which tolerates minor formatting differences as long as the final numeric answer is correct.

Performance

Baseline Result

Minitron-8B-Base can be deployed through vllm-ascend for online inference and benchmark evaluation.
Actual throughput and latency depend on hardware resources, prompt length, output length, concurrency, and runtime configuration.

Remarks

This document focuses on functional verification and benchmark accuracy on GSM8K.
Further benchmarking is recommended for:

  • request latency
  • throughput under concurrency
  • long-context inference
  • memory utilization
  • stability under continuous serving workloads