Adding a Multimodal Plugin

This document teaches you how to add a new modality to vLLM.

Each modality in vLLM is represented by a MultiModalPlugin and registered to MULTIMODAL_REGISTRY. For vLLM to recognize a new modality type, you have to create a new plugin and then pass it to register_plugin().

The remainder of this document details how to define custom MultiModalPlugin s.

Note

This article is a work in progress.