Skip to content

vllm_omni.diffusion.models.soulx_singer.modules.whisper_encoder

Frozen Whisper encoder wrapper (wav -> encoder embeddings).

WHISPER_MEL_FRAMES module-attribute 

WHISPER_MEL_FRAMES = 3000

audio module-attribute 

audio = torch.randn(1, 24000 * 25).float().to('cuda')

encoder module-attribute 

encoder = WhisperEncoder()

whisper_encoder_out module-attribute 

whisper_encoder_out = encoder.encode(audio, sr=24000)

WhisperEncoder

Auxiliary content encoder for SVC.

Kept in FP32 for stable HF Whisper inference under vLLM-Omni's default BF16 diffusion trunk (same idea as Stable Audio's FP32 VAE). Outputs are cast to the trunk dtype before fusion with the CFM conditioner.

device instance-attribute 

device = torch.device(
    device
    or ("cuda" if torch.cuda.is_available() else "cpu")
)

dtype property 

dtype: dtype

fe instance-attribute 

fe = WhisperFeatureExtractor.from_pretrained(
    "openai/whisper-base"
)

model instance-attribute 

model = self.model.to(
    device=self.device, dtype=torch.float32
)

encode 

encode(
    wav: Tensor,
    sr: int,
    *,
    output_dtype: dtype | None = None,
) -> Tensor

float 

float() -> WhisperEncoder