vllm_omni.transformers_utils.processors.ming ¶

Bases: ProcessorMixin

Top-level multimodal processor for Ming-flash-omni 2.0.

Adapted from Ming's BailingMM2Processor https://github.com/inclusionAI/Ming/blob/3954fcb880ff5e61ff128bcf7f1ec344d46a6fe3/processing_bailingmm2.py

Subprocessors include: - Qwen2VLImageProcessor (image/video) - MingWhisperFeatureExtractor (modified audio processor using Whisper's log-mel spectrogram)

attributes = [
    "image_processor",
    "audio_processor",
    "tokenizer",
]

audio_processor_class = 'AutoFeatureExtractor'

audio_token = PLACEHOLDER_AUDIO_TOKEN_IN_TEXT

chat_template = getattr(tokenizer, 'chat_template', None)

image_processor_class = 'AutoImageProcessor'

image_token = PLACEHOLDER_IMAGE_TOKEN_IN_TEXT

model_input_names

spatial_merge_size = merge_size

tokenizer_class = 'AutoTokenizer'

video_processor = video_processor

video_processor_class = 'AutoVideoProcessor'

video_token = PLACEHOLDER_VIDEO_TOKEN_IN_TEXT

apply_chat_template(
    conversation: list[dict[str, Any]],
    sys_prompt_exp: str | None = None,
    use_cot_system_prompt: bool = False,
    **kwargs,
) -> str

apply_system_template(
    sys_prompt_exp: str | None = None,
    use_cot_system_prompt: bool = False,
) -> str

batch_decode(*args, **kwargs)

decode(*args, **kwargs)

from_pretrained(
    pretrained_model_name_or_path, *args, **kwargs
)

save_pretrained(
    save_directory, push_to_hub: bool = False, **kwargs
)

Bases: FeatureExtractionMixin

Whisper log-mel feature extractor for Ming-flash-omni-2.0.

Produces audio_feats in the time-first packed format.

Adapted from Ming's WhisperAudioEncoder https://github.com/inclusionAI/Ming/blob/070dc3c13f95d97952ab7d22030df0c9e28a5122/modeling_whisper_encoder.py and HF transformers WhisperFeatureExtractor https://github.com/huggingface/transformers/blob/f842abaca95a7dbf3fc6e16122e7409109bc1431/src/transformers/models/whisper/feature_extraction_whisper.py#L33

feature_size = feature_size

model_input_names = ['audio_feats', 'audio_feats_lengths']

n_mels: int

sampling_rate = sampling_rate