Supported Models#

vLLM supports a variety of generative Transformer models in HuggingFace Transformers. The following is the list of model architectures that are currently supported by vLLM. Alongside each architecture, we include some popular models that use it.

Architecture

Models

Example HuggingFace Models

AquilaForCausalLM

Aquila

BAAI/Aquila-7B, BAAI/AquilaChat-7B, etc.

BaiChuanForCausalLM

Baichuan

baichuan-inc/Baichuan2-13B-Chat, baichuan-inc/Baichuan-7B, etc.

ChatGLMModel

ChatGLM

THUDM/chatglm2-6b, THUDM/chatglm3-6b, etc.

BloomForCausalLM

BLOOM, BLOOMZ, BLOOMChat

bigscience/bloom, bigscience/bloomz, etc.

FalconForCausalLM

Falcon

tiiuae/falcon-7b, tiiuae/falcon-40b, tiiuae/falcon-rw-7b, etc.

GPT2LMHeadModel

GPT-2

gpt2, gpt2-xl, etc.

GPTBigCodeForCausalLM

StarCoder, SantaCoder, WizardCoder

bigcode/starcoder, bigcode/gpt_bigcode-santacoder, WizardLM/WizardCoder-15B-V1.0, etc.

GPTJForCausalLM

GPT-J

EleutherAI/gpt-j-6b, nomic-ai/gpt4all-j, etc.

GPTNeoXForCausalLM

GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM

EleutherAI/gpt-neox-20b, EleutherAI/pythia-12b, OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5, databricks/dolly-v2-12b, stabilityai/stablelm-tuned-alpha-7b, etc.

InternLMForCausalLM

InternLM

internlm/internlm-7b, internlm/internlm-chat-7b, etc.

LlamaForCausalLM

LLaMA, LLaMA-2, Vicuna, Alpaca, Koala, Guanaco

meta-llama/Llama-2-13b-hf, meta-llama/Llama-2-70b-hf, openlm-research/open_llama_13b, lmsys/vicuna-13b-v1.3, young-geng/koala, etc.

MistralForCausalLM

Mistral, Mistral-Instruct

mistralai/Mistral-7B-v0.1, mistralai/Mistral-7B-Instruct-v0.1, etc.

MPTForCausalLM

MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter

mosaicml/mpt-7b, mosaicml/mpt-7b-storywriter, mosaicml/mpt-30b, etc.

OPTForCausalLM

OPT, OPT-IML

facebook/opt-66b, facebook/opt-iml-max-30b, etc.

PhiForCausalLM

Phi-1.5

microsoft/phi-1_5, etc.

QWenLMHeadModel

Qwen

Qwen/Qwen-7B, Qwen/Qwen-7B-Chat, etc.

YiForCausalLM

Yi

01-ai/Yi-6B, 01-ai/Yi-34B, etc.

If your model uses one of the above model architectures, you can seamlessly run your model with vLLM. Otherwise, please refer to Adding a New Model for instructions on how to implement support for your model. Alternatively, you can raise an issue on our GitHub project.

Tip

The easiest way to check if your model is supported is to run the program below:

from vllm import LLM

llm = LLM(model=...)  # Name or path of your model
output = llm.generate("Hello, my name is")
print(output)

To use model from www.modelscope.cn

$ export VLLM_USE_MODELSCOPE=True
from vllm import LLM

llm = LLM(model=..., revision=..., trust_remote_code=True)  # Name or path of your model
output = llm.generate("Hello, my name is")
print(output)

If vLLM successfully generates text, it indicates that your model is supported.