Supported Models

vLLM supports a variety of generative and embedding models from HuggingFace (HF) Transformers. This page lists the model architectures that are currently supported by vLLM. Alongside each architecture, we include some popular models that use it.

For other models, you can check the config.json file inside the model repository. If the "architectures" field contains a model architecture listed below, then it should be supported in theory.

Tip

The easiest way to check if your model is really supported at runtime is to run the program below:

from vllm import LLM

llm = LLM(model=...)  # Name or path of your model
output = llm.generate("Hello, my name is")
print(output)

If vLLM successfully generates text, it indicates that your model is supported.

Otherwise, please refer to Adding a New Model and Enabling Multimodal Inputs for instructions on how to implement your model in vLLM. Alternatively, you can open an issue on GitHub to request vLLM support.

Note

To use models from ModelScope instead of HuggingFace Hub, set an environment variable:

$ export VLLM_USE_MODELSCOPE=True

And use with trust_remote_code=True.

from vllm import LLM

llm = LLM(model=..., revision=..., trust_remote_code=True)  # Name or path of your model
output = llm.generate("Hello, my name is")
print(output)

Text-only Language Models

Text Generation

Architecture	Models	Example HF Models	LoRA	PP
AquilaForCausalLM	Aquila, Aquila2	BAAI/Aquila-7B, BAAI/AquilaChat-7B, etc.	✅︎	✅︎
ArcticForCausalLM	Arctic	Snowflake/snowflake-arctic-base, Snowflake/snowflake-arctic-instruct, etc.		✅︎
BaiChuanForCausalLM	Baichuan2, Baichuan	baichuan-inc/Baichuan2-13B-Chat, baichuan-inc/Baichuan-7B, etc.	✅︎	✅︎
BloomForCausalLM	BLOOM, BLOOMZ, BLOOMChat	bigscience/bloom, bigscience/bloomz, etc.		✅︎
BartForConditionalGeneration	BART	facebook/bart-base, facebook/bart-large-cnn, etc.
ChatGLMModel	ChatGLM	THUDM/chatglm2-6b, THUDM/chatglm3-6b, etc.	✅︎	✅︎
CohereForCausalLM	Command-R	CohereForAI/c4ai-command-r-v01, etc.	✅︎	✅︎
DbrxForCausalLM	DBRX	databricks/dbrx-base, databricks/dbrx-instruct, etc.		✅︎
DeciLMForCausalLM	DeciLM	Deci/DeciLM-7B, Deci/DeciLM-7B-instruct, etc.		✅︎
DeepseekForCausalLM	DeepSeek	deepseek-ai/deepseek-llm-67b-base, deepseek-ai/deepseek-llm-7b-chat etc.		✅︎
DeepseekV2ForCausalLM	DeepSeek-V2	deepseek-ai/DeepSeek-V2, deepseek-ai/DeepSeek-V2-Chat etc.		✅︎
ExaoneForCausalLM	EXAONE-3	LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct, etc.	✅︎	✅︎
FalconForCausalLM	Falcon	tiiuae/falcon-7b, tiiuae/falcon-40b, tiiuae/falcon-rw-7b, etc.		✅︎
FalconMambaForCausalLM	FalconMamba	tiiuae/falcon-mamba-7b, tiiuae/falcon-mamba-7b-instruct, etc.	✅︎
GemmaForCausalLM	Gemma	google/gemma-2b, google/gemma-7b, etc.	✅︎	✅︎
Gemma2ForCausalLM	Gemma2	google/gemma-2-9b, google/gemma-2-27b, etc.	✅︎	✅︎
GlmForCausalLM	GLM-4	THUDM/glm-4-9b-chat-hf, etc.	✅︎	✅︎
GPT2LMHeadModel	GPT-2	gpt2, gpt2-xl, etc.		✅︎
GPTBigCodeForCausalLM	StarCoder, SantaCoder, WizardCoder	bigcode/starcoder, bigcode/gpt_bigcode-santacoder, WizardLM/WizardCoder-15B-V1.0, etc.	✅︎	✅︎
GPTJForCausalLM	GPT-J	EleutherAI/gpt-j-6b, nomic-ai/gpt4all-j, etc.		✅︎
GPTNeoXForCausalLM	GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM	EleutherAI/gpt-neox-20b, EleutherAI/pythia-12b, OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5, databricks/dolly-v2-12b, stabilityai/stablelm-tuned-alpha-7b, etc.		✅︎
GraniteForCausalLM	Granite 3.0, PowerLM	ibm-granite/granite-3.0-2b-base, ibm-granite/granite-3.0-8b-instruct, ibm/PowerLM-3b, etc.	✅︎	✅︎
GraniteMoeForCausalLM	Granite 3.0 MoE, PowerMoE	ibm-granite/granite-3.0-1b-a400m-base, ibm-granite/granite-3.0-3b-a800m-instruct, ibm/PowerMoE-3b, etc.	✅︎	✅︎
InternLMForCausalLM	InternLM	internlm/internlm-7b, internlm/internlm-chat-7b, etc.	✅︎	✅︎
InternLM2ForCausalLM	InternLM2	internlm/internlm2-7b, internlm/internlm2-chat-7b, etc.	✅︎	✅︎
JAISLMHeadModel	Jais	inceptionai/jais-13b, inceptionai/jais-13b-chat, inceptionai/jais-30b-v3, inceptionai/jais-30b-chat-v3, etc.		✅︎
JambaForCausalLM	Jamba	ai21labs/AI21-Jamba-1.5-Large, ai21labs/AI21-Jamba-1.5-Mini, ai21labs/Jamba-v0.1, etc.	✅︎
LlamaForCausalLM	Llama 3.1, Llama 3, Llama 2, LLaMA, Yi	meta-llama/Meta-Llama-3.1-405B-Instruct, meta-llama/Meta-Llama-3.1-70B, meta-llama/Meta-Llama-3-70B-Instruct, meta-llama/Llama-2-70b-hf, 01-ai/Yi-34B, etc.	✅︎	✅︎
MambaForCausalLM	Mamba	state-spaces/mamba-130m-hf, state-spaces/mamba-790m-hf, state-spaces/mamba-2.8b-hf, etc.
MiniCPMForCausalLM	MiniCPM	openbmb/MiniCPM-2B-sft-bf16, openbmb/MiniCPM-2B-dpo-bf16, openbmb/MiniCPM-S-1B-sft, etc.	✅︎	✅︎
MiniCPM3ForCausalLM	MiniCPM3	openbmb/MiniCPM3-4B, etc.	✅︎	✅︎
MistralForCausalLM	Mistral, Mistral-Instruct	mistralai/Mistral-7B-v0.1, mistralai/Mistral-7B-Instruct-v0.1, etc.	✅︎	✅︎
MixtralForCausalLM	Mixtral-8x7B, Mixtral-8x7B-Instruct	mistralai/Mixtral-8x7B-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1, mistral-community/Mixtral-8x22B-v0.1, etc.	✅︎	✅︎
MPTForCausalLM	MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter	mosaicml/mpt-7b, mosaicml/mpt-7b-storywriter, mosaicml/mpt-30b, etc.		✅︎
NemotronForCausalLM	Nemotron-3, Nemotron-4, Minitron	nvidia/Minitron-8B-Base, mgoin/Nemotron-4-340B-Base-hf-FP8, etc.	✅︎	✅︎
OLMoForCausalLM	OLMo	allenai/OLMo-1B-hf, allenai/OLMo-7B-hf, etc.		✅︎
OLMo2ForCausalLM	OLMo2	allenai/OLMo2-7B-1124, etc.		✅︎
OLMoEForCausalLM	OLMoE	allenai/OLMoE-1B-7B-0924, allenai/OLMoE-1B-7B-0924-Instruct, etc.	✅︎	✅︎
OPTForCausalLM	OPT, OPT-IML	facebook/opt-66b, facebook/opt-iml-max-30b, etc.		✅︎
OrionForCausalLM	Orion	OrionStarAI/Orion-14B-Base, OrionStarAI/Orion-14B-Chat, etc.		✅︎
PhiForCausalLM	Phi	microsoft/phi-1_5, microsoft/phi-2, etc.	✅︎	✅︎
Phi3ForCausalLM	Phi-3	microsoft/Phi-3-mini-4k-instruct, microsoft/Phi-3-mini-128k-instruct, microsoft/Phi-3-medium-128k-instruct, etc.	✅︎	✅︎
Phi3SmallForCausalLM	Phi-3-Small	microsoft/Phi-3-small-8k-instruct, microsoft/Phi-3-small-128k-instruct, etc.		✅︎
PhiMoEForCausalLM	Phi-3.5-MoE	microsoft/Phi-3.5-MoE-instruct, etc.	✅︎	✅︎
PersimmonForCausalLM	Persimmon	adept/persimmon-8b-base, adept/persimmon-8b-chat, etc.		✅︎
QWenLMHeadModel	Qwen	Qwen/Qwen-7B, Qwen/Qwen-7B-Chat, etc.	✅︎	✅︎
Qwen2ForCausalLM	Qwen2	Qwen/Qwen2-7B-Instruct, Qwen/Qwen2-7B, etc.	✅︎	✅︎
Qwen2MoeForCausalLM	Qwen2MoE	Qwen/Qwen1.5-MoE-A2.7B, Qwen/Qwen1.5-MoE-A2.7B-Chat, etc.		✅︎
StableLmForCausalLM	StableLM	stabilityai/stablelm-3b-4e1t, stabilityai/stablelm-base-alpha-7b-v2, etc.		✅︎
Starcoder2ForCausalLM	Starcoder2	bigcode/starcoder2-3b, bigcode/starcoder2-7b, bigcode/starcoder2-15b, etc.		✅︎
SolarForCausalLM	Solar Pro	upstage/solar-pro-preview-instruct, etc.	✅︎	✅︎
TeleChat2ForCausalLM	TeleChat2	TeleAI/TeleChat2-3B, TeleAI/TeleChat2-7B, TeleAI/TeleChat2-35B, etc.	✅︎	✅︎
XverseForCausalLM	XVERSE	xverse/XVERSE-7B-Chat, xverse/XVERSE-13B-Chat, xverse/XVERSE-65B-Chat, etc.	✅︎	✅︎

Note

Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.

Text Embedding

Architecture	Models	Example HF Models	LoRA	PP
BertModel	BERT-based	BAAI/bge-base-en-v1.5, etc.
Gemma2Model	Gemma2-based	BAAI/bge-multilingual-gemma2, etc.		✅︎
LlamaModel, LlamaForCausalLM, MistralModel, etc.	Llama-based	intfloat/e5-mistral-7b-instruct, etc.	✅︎	✅︎
Qwen2Model, Qwen2ForCausalLM	Qwen2-based	ssmits/Qwen2-7B-Instruct-embed-base (see note), Alibaba-NLP/gte-Qwen2-7B-instruct (see note), etc.	✅︎	✅︎
RobertaModel, RobertaForMaskedLM	RoBERTa-based	sentence-transformers/all-roberta-large-v1, sentence-transformers/all-roberta-large-v1, etc.
XLMRobertaModel	XLM-RoBERTa-based	intfloat/multilingual-e5-large, etc.

Important

Some model architectures support both generation and embedding tasks. In this case, you have to pass --task embedding to run the model in embedding mode.

Tip

You can override the model’s pooling method by passing --override-pooler-config.

Note

ssmits/Qwen2-7B-Instruct-embed-base has an improperly defined Sentence Transformers config. You should manually set mean pooling by passing --override-pooler-config '{"pooling_type": "MEAN"}'.

Note

Unlike base Qwen2, Alibaba-NLP/gte-Qwen2-7B-instruct uses bi-directional attention. You can set --hf-overrides '{"is_causal": false}' to change the attention mask accordingly.

On the other hand, its 1.5B variant (Alibaba-NLP/gte-Qwen2-1.5B-instruct) uses causal attention despite being described otherwise on its model card.

Reward Modeling

Architecture	Models	Example HF Models	LoRA	PP
LlamaForCausalLM	Llama-based	peiyi9979/math-shepherd-mistral-7b-prm, etc.	✅︎	✅︎
Qwen2ForRewardModel	Qwen2-based	Qwen/Qwen2.5-Math-RM-72B, etc.	✅︎	✅︎

Important

For process-supervised reward models such as peiyi9979/math-shepherd-mistral-7b-prm, the pooling config should be set explicitly, e.g.: --override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'.

Note

As an interim measure, these models are supported in both offline and online inference via Embeddings API.

Classification

Architecture	Models	Example HF Models	LoRA	PP
Qwen2ForSequenceClassification	Qwen2-based	jason9693/Qwen2.5-1.5B-apeach, etc.	✅︎	✅︎

Note

As an interim measure, these models are supported in both offline and online inference via Embeddings API.

Sentence Pair Scoring

Architecture	Models	Example HF Models	LoRA	PP
BertForSequenceClassification	BERT-based	cross-encoder/ms-marco-MiniLM-L-6-v2, etc.
RobertaForSequenceClassification	RoBERTa-based	cross-encoder/quora-roberta-base, etc.
XLMRobertaForSequenceClassification	XLM-RoBERTa-based	BAAI/bge-reranker-v2-m3, etc.

Note

These models are supported in both offline and online inference via Score API.

Multimodal Language Models

The following modalities are supported depending on the model:

• Text

• Image

• Video

• Audio

Any combination of modalities joined by + are supported.

• e.g.: T + I means that the model supports text-only, image-only, and text-with-image inputs.

On the other hand, modalities separated by / are mutually exclusive.

• e.g.: T / I means that the model supports text-only and image-only inputs, but not text-with-image inputs.

Text Generation

Architecture	Models	Inputs	Example HF Models	LoRA	PP
AriaForConditionalGeneration	Aria	T + I	rhymes-ai/Aria		✅︎
Blip2ForConditionalGeneration	BLIP-2	T + IE	Salesforce/blip2-opt-2.7b, Salesforce/blip2-opt-6.7b, etc.		✅︎
ChameleonForConditionalGeneration	Chameleon	T + I	facebook/chameleon-7b etc.		✅︎
FuyuForCausalLM	Fuyu	T + I	adept/fuyu-8b etc.		✅︎
ChatGLMModel	GLM-4V	T + I	THUDM/glm-4v-9b etc.	✅︎	✅︎
H2OVLChatModel	H2OVL	T + IE+	h2oai/h2ovl-mississippi-800m, h2oai/h2ovl-mississippi-2b, etc.		✅︎
Idefics3ForConditionalGeneration	Idefics3	T + I	HuggingFaceM4/Idefics3-8B-Llama3 etc.	✅︎
InternVLChatModel	InternVL2	T + IE+	OpenGVLab/Mono-InternVL-2B, OpenGVLab/InternVL2-4B, OpenGVLab/InternVL2-8B, etc.		✅︎
LlavaForConditionalGeneration	LLaVA-1.5	T + IE+	llava-hf/llava-1.5-7b-hf, llava-hf/llava-1.5-13b-hf, etc.		✅︎
LlavaNextForConditionalGeneration	LLaVA-NeXT	T + IE+	llava-hf/llava-v1.6-mistral-7b-hf, llava-hf/llava-v1.6-vicuna-7b-hf, etc.		✅︎
LlavaNextVideoForConditionalGeneration	LLaVA-NeXT-Video	T + V	llava-hf/LLaVA-NeXT-Video-7B-hf, etc.		✅︎
LlavaOnevisionForConditionalGeneration	LLaVA-Onevision	T + I+ + V+	llava-hf/llava-onevision-qwen2-7b-ov-hf, llava-hf/llava-onevision-qwen2-0.5b-ov-hf, etc.		✅︎
MiniCPMV	MiniCPM-V	T + IE+	openbmb/MiniCPM-V-2 (see note), openbmb/MiniCPM-Llama3-V-2_5, openbmb/MiniCPM-V-2_6, etc.	✅︎	✅︎
MllamaForConditionalGeneration	Llama 3.2	T + I+	meta-llama/Llama-3.2-90B-Vision-Instruct, meta-llama/Llama-3.2-11B-Vision, etc.
MolmoForCausalLM	Molmo	T + I	allenai/Molmo-7B-D-0924, allenai/Molmo-72B-0924, etc.		✅︎
NVLM_D_Model	NVLM-D 1.0	T + IE+	nvidia/NVLM-D-72B, etc.		✅︎
PaliGemmaForConditionalGeneration	PaliGemma	T + IE	google/paligemma-3b-pt-224, google/paligemma-3b-mix-224, etc.		✅︎
Phi3VForCausalLM	Phi-3-Vision, Phi-3.5-Vision	T + IE+	microsoft/Phi-3-vision-128k-instruct, microsoft/Phi-3.5-vision-instruct etc.		✅︎
PixtralForConditionalGeneration	Pixtral	T + I+	mistralai/Pixtral-12B-2409, mistral-community/pixtral-12b etc.		✅︎
QWenLMHeadModel	Qwen-VL	T + IE+	Qwen/Qwen-VL, Qwen/Qwen-VL-Chat, etc.	✅︎	✅︎
Qwen2AudioForConditionalGeneration	Qwen2-Audio	T + A+	Qwen/Qwen2-Audio-7B-Instruct		✅︎
Qwen2VLForConditionalGeneration	Qwen2-VL	T + IE+ + VE+	Qwen/Qwen2-VL-2B-Instruct, Qwen/Qwen2-VL-7B-Instruct, Qwen/Qwen2-VL-72B-Instruct, etc.	✅︎	✅︎
UltravoxModel	Ultravox	T + AE+	fixie-ai/ultravox-v0_3		✅︎

^E Pre-computed embeddings can be inputted for this modality.

⁺ Multiple items can be inputted per text prompt for this modality.

Important

To enable multiple multi-modal items per text prompt, you have to set limit_mm_per_prompt (offline inference) or --limit-mm-per-prompt (online inference). For example, to enable passing up to 4 images per text prompt:

llm = LLM(
    model="Qwen/Qwen2-VL-7B-Instruct",
    limit_mm_per_prompt={"image": 4},
)

vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=4

Note

vLLM currently only supports adding LoRA to the language backbone of multimodal models.

Note

The official openbmb/MiniCPM-V-2 doesn’t work yet, so we need to use a fork (HwwwH/MiniCPM-V-2) for now. For more details, please see: vllm-project/vllm#4087

Multimodal Embedding

Architecture	Models	Inputs	Example HF Models	LoRA	PP
LlavaNextForConditionalGeneration	LLaVA-NeXT-based	T / I	royokong/e5-v		✅︎
Phi3VForCausalLM	Phi-3-Vision-based	T + I	TIGER-Lab/VLM2Vec-Full	🚧	✅︎
Qwen2VLForConditionalGeneration	Qwen2-VL-based	T + I	MrLight/dse-qwen2-2b-mrl-v1		✅︎

Important

Some model architectures support both generation and embedding tasks. In this case, you have to pass --task embedding to run the model in embedding mode.

Tip

You can override the model’s pooling method by passing --override-pooler-config.

Model Support Policy

At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:

1. Community-Driven Support: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. Call for contribution: PRs coming directly from model vendors are greatly appreciated!

2. Best-Effort Consistency: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.

Tip

When comparing the output of model.generate from HuggingFace Transformers with the output of llm.generate from vLLM, note that the former reads the model’s generation config file (i.e., generation_config.json) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.

3. Issue Resolution and Model Updates: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.

4. Monitoring and Updates: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.

5. Selective Focus: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.

Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.

Note that, as an inference engine, vLLM does not introduce new models. Therefore, all models supported by vLLM are third-party models in this regard.

We have the following levels of testing for models:

1. Strict Consistency: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to models tests for the models that have passed this test.

2. Output Sensibility: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.

3. Runtime Functionality: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to functionality tests and examples for the models that have passed this test.

4. Community Feedback: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.