LoRA Adapters Guide¶
Overview¶
Like vLLM, vllm-ascend supports LoRA as well. The usage and more details can be found in vLLM official document.
You can refer to Supported Models to find which models support LoRA in vLLM.
You can run LoRA with ACLGraph mode now. Please refer to Graph Mode Guide for better LoRA performance.
Address for downloading models:
- base model: https://www.modelscope.cn/models/vllm-ascend/Llama-2-7b-hf/files
- loRA model: https://www.modelscope.cn/models/vllm-ascend/llama-2-7b-sql-lora-test/files
Example¶
We provide a simple LoRA example here, which enables the ACLGraph mode by default.
vllm serve meta-llama/Llama-2-7b \
--enable-lora \
--lora-modules '{"name": "sql-lora", "path": "/path/to/lora", "base_model_name": "meta-llama/Llama-2-7b"}'
Note¶
- We have implemented LoRA-related AscendC operators, such as bgmv_shrink, bgmv_expand, sgmv_shrink and sgmv_expand. You can find them under the
csrc/kernelsdirectory of vllm-ascend repo.
LoRA is supported for both dense and mixture-of-experts (MoE) models. The current MoE support status is as follows:
| MoE mode | Tensor parallel (AllGather) | Expert parallel (All-to-All) |
|---|---|---|
| Non-quantized | Supported | Supported |
| W8A8 dynamic quantization | Supported | Supported |
Other MoE quantization methods, Fused MC2, and dynamic EPLB are not supported with LoRA.