Registering a Model to vLLM

vLLM relies on a model registry to determine how to run each model. A list of pre-registered architectures can be found here.

If your model is not on this list, you must register it to vLLM. This page provides detailed instructions on how to do so.

Built-in models

To add a model directly to the vLLM library, start by forking our GitHub repository and then build it from source. This gives you the ability to modify the codebase and test your model.

After you have implemented your model (see tutorial), put it into the vllm/model_executor/models directory. Then, add your model class to _VLLM_MODELS in vllm/model_executor/models/registry.py so that it is automatically registered upon importing vLLM. Finally, update our list of supported models to promote your model!

Important

The list of models in each section should be maintained in alphabetical order.

Out-of-tree models

You can load an external model using a plugin without modifying the vLLM codebase.

See also

vLLM’s Plugin System

To register the model, use the following code:

from vllm import ModelRegistry
from your_code import YourModelForCausalLM
ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)

If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like RuntimeError: Cannot re-initialize CUDA in forked subprocess:

from vllm import ModelRegistry

ModelRegistry.register_model("YourModelForCausalLM", "your_code:YourModelForCausalLM")

Important

If your model is a multimodal model, ensure the model class implements the SupportsMultiModal interface. Read more about that here.

Note

Although you can directly put these code snippets in your script using vllm.LLM, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server.