Data Processing

Module Contents

vllm.multimodal.processing.PromptSeq

A token sequence (list of token IDs) or text.

alias of Union[str, list[int]]

class vllm.multimodal.processing.PromptReplacementDetails(full: str | list[int], features: str | list[int])

Details about the replacement token sequence or text.

full: str | list[int]

The full replacement.

features: str | list[int]

The part of the replacement that corresponds to feature placeholders; this will be replaced by the output of the vision encoder during model inference.

vllm.multimodal.processing.PromptRepl

The replacement token sequence or text.

If only part of the replacement corresponds to feature placeholders, you can use PromptReplacementDetails to specify which part.

alias of Union[str, list[int], PromptReplacementDetails]

class vllm.multimodal.processing.PromptReplacement(modality: str, target: str | list[int], replacement: Callable[[int], str | list[int] | PromptReplacementDetails] | str | list[int] | PromptReplacementDetails)

Defines how to replace portions of an input prompt with placeholder tokens.

Example

For each image, replace one <image> input placeholder in the prompt with a number of <image> feature placeholders equal to the feature size of the vision encoder:

PromptReplacement(
    modality="image",
    target="<image>",
    replacement="<image>" * image_feature_size,
)

As above, but further pad the feature placeholders with <image_bos> and <image_eos>`, which are not supposed to be passed to the vision encoder:

PromptReplacement(
    modality="image",
    target="<image>",
    replacement=PromptReplacementDetails(
        full="".join([
            "<image_bos>",
            "<image>" * image_feature_size,
            "<image_eos>",
        ]),
        features="<image>" * image_feature_size,
    ),
)

To avoid unnecessary tokenization during prompt replacement, we recommended passing token sequences instead of text:

PromptReplacement(
    modality="image",
    target=[image_token_id],
    replacement=PromptReplacementDetails(
        full=([image_bos_id] + [image_token_id] * image_feature_size
              + [image_eos_id]),
        features=[image_token_id] * image_feature_size,
    ),
)

modality: str

The modality for which the replacement is made.

target: str | list[int]

The token sequence (or text) to find and replace.

replacement: Callable[[int], str | list[int] | PromptReplacementDetails] | str | list[int] | PromptReplacementDetails

Given the index of the processed item within modality, output the replacement token sequence (or text).

For convenience, you can directly pass in the replacement token sequence (or text) instead of a function if it does not depend on the input.

vllm.multimodal.processing.full_groupby_modality(values: Iterable[_M]) → ItemsView[str, list[_M]]

Convenience function to apply full_groupby() based on modality.

class vllm.multimodal.processing.BoundPromptReplacement(tokenizer: transformers.PreTrainedTokenizer | transformers.PreTrainedTokenizerFast | MistralTokenizer, modality: str, _target: str | list[int], _replacement: Callable[[int], str | list[int] | PromptReplacementDetails] | str | list[int] | PromptReplacementDetails)

A PromptReplacement bound to a tokenizer to automatically convert target and the result of get_replacement() between token sequence and text representations.

property target: _BoundPromptSequence

The token sequence (or text) to find and replace.

get_replacement(item_idx: int) → _BoundPromptReplacementGroup

Given the index of the processed item within modality, output the replacement token sequence (or text).

vllm.multimodal.processing.iter_token_matches(token_ids: list[int], match_ids: list[int]) → Generator[_TokenMatch]

Yield each occurrence of match_ids in token_ids.

Note that empty matches are ignored.

class vllm.multimodal.processing.PlaceholderFeaturesInfo(modality: str, item_idx: int, start_idx: int, tokens: list[int])

vllm.multimodal.processing.find_token_matches(prompt: list[int], prompt_repls: Sequence[BoundPromptReplacement]) → list[vllm.multimodal.processing._PromptReplacementTokenMatch]

Return each target of prompt_repls found in prompt.

vllm.multimodal.processing.find_text_matches(prompt: str, prompt_repls: Sequence[BoundPromptReplacement]) → list[vllm.multimodal.processing._PromptReplacementTextMatch]

Return each target of prompt_repls found in prompt.

vllm.multimodal.processing.replace_token_matches(prompt: list[int], mm_matches: Mapping[str, Sequence[_PromptReplacementTokenMatch]], mm_item_counts: Mapping[str, int]) → list[int]

Apply the replacements in mm_matches to prompt.

vllm.multimodal.processing.replace_text_matches(prompt: str, mm_matches: Mapping[str, Sequence[_PromptReplacementTextMatch]], mm_item_counts: Mapping[str, int]) → str

Apply the replacements in mm_matches to prompt.

class vllm.multimodal.processing.BaseProcessingInfo(ctx: InputProcessingContext)

Base class to provide the information necessary for data processing.

get_hf_processor(**kwargs: object) → transformers.ProcessorMixin

Subclasses can override this method to handle specific kwargs from model config or user inputs.

abstract get_supported_mm_limits() → Mapping[str, int | None]

Return the maximum supported number of items for each modality.

A value of None means unlimited number of items.

Omitting a modality from the returned dictionary means that it is not supported at all.

abstract get_mm_max_tokens_per_item(seq_len: int) → Mapping[str, int]

Get the maximum possible number of tokens per data item for each modality.

The dictionary returned by this method should have the same keys as that returned by get_supported_mm_limits().

class vllm.multimodal.processing.BaseMultiModalProcessor(info: _I, dummy_inputs: BaseDummyInputsBuilder[_I], *, cache: ProcessingCache | None = None, enable_sanity_checks: bool = True)

Abstract base class to process multi-modal inputs to be used in vLLM.

Not to be confused with transformers.ProcessorMixin.

apply(prompt: str | list[int], mm_data: Mapping[str, Any | list[Any]], hf_processor_mm_kwargs: Mapping[str, object]) → MultiModalInputs

Process multi-modal inputs to be used in vLLM.

The main steps are:

1. Apply HF Processor on prompt text and multi-modal data together, outputting token IDs and processed tensors.

2. Find and replace sequences in the token IDs with placeholder tokens. The number of placeholder tokens equals the feature size of the multi-modal data outputted by the multi-modal encoder.

3. Extract information about the placeholder tokens from the processed token IDs.