Skip to content

vllm_omni.diffusion.models.soulx_singer.modules.preprocess.rmvpe

RMVPE F0 extractor.

All model code is inlined (originally from RickyL-2000/ROSVOT modules/pe/rmvpe/) with two minor substitutions:

  • librosa.filters.mel → torchaudio.functional.melscale_fbanks (already used elsewhere in vllm-omni);
  • librosa.sequence.viterbi → hand-written numpy viterbi (non-default path).
  • pyworld (deprecated) → removed.

RMVPE_SAMPLE_RATE module-attribute 

RMVPE_SAMPLE_RATE = 16000

logger module-attribute 

logger = init_logger(__name__)

BiGRU

Bases: Module

gru instance-attribute 

gru = nn.GRU(
    input_features,
    hidden_features,
    num_layers=num_layers,
    batch_first=True,
    bidirectional=True,
)

forward 

forward(x: Tensor) -> Tensor

ConvBlockRes

Bases: Module

conv instance-attribute 

conv = nn.Sequential(
    nn.Conv2d(
        in_channels,
        out_channels,
        kernel_size=(3, 3),
        stride=(1, 1),
        padding=(1, 1),
        bias=False,
    ),
    nn.BatchNorm2d(out_channels, momentum=momentum),
    nn.ReLU(),
    nn.Conv2d(
        out_channels,
        out_channels,
        kernel_size=(3, 3),
        stride=(1, 1),
        padding=(1, 1),
        bias=False,
    ),
    nn.BatchNorm2d(out_channels, momentum=momentum),
    nn.ReLU(),
)

shortcut instance-attribute 

shortcut = nn.Conv2d(in_channels, out_channels, (1, 1))

forward 

forward(x: Tensor) -> Tensor

Decoder

Bases: Module

layers instance-attribute 

layers = nn.ModuleList()

n_decoders instance-attribute 

n_decoders = n_decoders

forward 

forward(x: Tensor, concat_tensors: list[Tensor]) -> Tensor

DeepUnet0

Bases: Module

decoder instance-attribute 

decoder = Decoder(
    self.encoder.out_channel,
    en_de_layers,
    kernel_size,
    n_blocks,
)

encoder instance-attribute 

encoder = Encoder(
    in_channels,
    _N_MELS,
    en_de_layers,
    kernel_size,
    n_blocks,
    en_out_channels,
)

intermediate instance-attribute 

intermediate = Intermediate(
    self.encoder.out_channel // 2,
    self.encoder.out_channel,
    inter_layers,
    n_blocks,
)

tf instance-attribute 

tf = TimbreFilter(self.encoder.latent_channels)

forward 

forward(x: Tensor) -> Tensor

E2E0

Bases: Module

E2E0 pitch estimator network (DeepUnet0 + CNN + BiGRU + classifier).

cnn instance-attribute 

cnn = nn.Conv2d(en_out_channels, 3, (3, 3), padding=(1, 1))

fc instance-attribute 

fc = nn.Sequential(
    BiGRU(3 * _N_MELS, 256, n_gru),
    nn.Linear(512, _N_CLASS),
    nn.Dropout(0.25),
    nn.Sigmoid(),
)

unet instance-attribute 

unet = DeepUnet0(
    kernel_size,
    n_blocks,
    en_de_layers,
    inter_layers,
    in_channels,
    en_out_channels,
)

forward 

forward(mel: Tensor) -> Tensor

Encoder

Bases: Module

bn instance-attribute 

bn = nn.BatchNorm2d(in_channels, momentum=momentum)

latent_channels instance-attribute 

latent_channels: list[list[int]] = []

layers instance-attribute 

layers = nn.ModuleList()

n_encoders instance-attribute 

n_encoders = n_encoders

out_channel instance-attribute 

out_channel = out_channels

out_size instance-attribute 

out_size = in_size

forward 

forward(x: Tensor) -> tuple[Tensor, list[Tensor]]

Intermediate

Bases: Module

layers instance-attribute 

layers = nn.ModuleList()

n_inters instance-attribute 

n_inters = n_inters

forward 

forward(x: Tensor) -> Tensor

MelSpectrogram

Bases: Module

Mel-spectrogram extractor matching ROSVOT's MelSpectrogram.

Log-mel spectrogram via torch.stft + adaptive filterbank.

clamp instance-attribute 

clamp = clamp

hann_window instance-attribute 

hann_window: dict[str, Tensor] = {}

hop_length instance-attribute 

hop_length = hop_length

n_fft instance-attribute 

n_fft = n_fft

n_mel_channels instance-attribute 

n_mel_channels = n_mel_channels

sampling_rate instance-attribute 

sampling_rate = sampling_rate

win_length instance-attribute 

win_length = win_length

forward 

forward(
    audio: Tensor,
    keyshift: float = 0,
    speed: float = 1,
    center: bool = True,
) -> Tensor

RMVPE

Bases: Module, SupportAudioInput, SupportsComponentDiscovery

RMVPE F0 extractor (SoulX-Singer).

Wraps the E2E0 model with mel-spectrogram front-end and decoding post-processing.

device instance-attribute 

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

hop_length instance-attribute 

hop_length = hop_length

mel_extractor instance-attribute 

mel_extractor = MelSpectrogram(
    _N_MELS,
    RMVPE_SAMPLE_RATE,
    _WINDOW_LENGTH,
    hop_length,
    None,
    _MEL_FMIN,
    _MEL_FMAX,
).to(self.device)

model instance-attribute 

model = self.model.to(self.device)

resample_kernel instance-attribute 

resample_kernel: dict[str, Resample] = {}

support_audio_input class-attribute 

support_audio_input: bool = True

decode 

decode(
    hidden: Tensor,
    thread: float = 0.03,
    use_viterbi: bool = False,
) -> ndarray

get_pitch 

get_pitch(
    waveform,
    sample_rate: int,
    hop_size: int,
    length: int,
    interp_uv: bool = False,
    fmin: float = 50,
    fmax: float = 1000,
) -> tuple[ndarray, ndarray]

get_pitch_batch 

get_pitch_batch(
    waveforms,
    sample_rate: int,
    hop_size: int,
    lengths: list[int],
    interp_uv: bool = False,
    fmin: float = 50,
    fmax: float = 1000,
) -> tuple[list[ndarray], list[ndarray]]

infer_from_audio 

infer_from_audio(
    audio: ndarray,
    sample_rate: int = 16000,
    thread: float = 0.03,
    use_viterbi: bool = False,
) -> ndarray

infer_from_audio_batch 

infer_from_audio_batch(
    audios,
    sample_rate: int = 16000,
    thread: float = 0.03,
    use_viterbi: bool = False,
) -> list[ndarray]

mel2hidden 

mel2hidden(mel: Tensor) -> Tensor

postprocess 

postprocess(
    f0: ndarray,
    fmin: float = 50,
    fmax: float = 1000,
    min_gap: int = 2,
) -> ndarray

release_cuda 

release_cuda()

ResDecoderBlock

Bases: Module

conv1 instance-attribute 

conv1 = nn.Sequential(
    nn.ConvTranspose2d(
        in_channels,
        out_channels,
        kernel_size=(3, 3),
        stride=stride,
        padding=(1, 1),
        output_padding=out_padding,
        bias=False,
    ),
    nn.BatchNorm2d(out_channels, momentum=momentum),
    nn.ReLU(),
)

conv2 instance-attribute 

conv2 = nn.ModuleList()

n_blocks instance-attribute 

n_blocks = n_blocks

forward 

forward(x: Tensor, concat_tensor: Tensor) -> Tensor

ResEncoderBlock

Bases: Module

conv instance-attribute 

conv = nn.ModuleList()

kernel_size instance-attribute 

kernel_size = kernel_size

n_blocks instance-attribute 

n_blocks = n_blocks

pool instance-attribute 

pool = nn.AvgPool2d(kernel_size=kernel_size)

forward 

forward(x: Tensor) -> Tensor | tuple[Tensor, Tensor]

TimbreFilter

Bases: Module

layers instance-attribute 

layers = nn.ModuleList()

forward 

forward(x_tensors: list[Tensor]) -> list[Tensor]

numpy_viterbi 

numpy_viterbi(
    prob: ndarray, transition: ndarray
) -> ndarray

Viterbi decoding on log-probabilities (numpy).

Parameters:

Name Type Description Default
prob ndarray

Emission probabilities, shape (n_states, n_obs).

 required
transition ndarray

Transition matrix, shape (n_states, n_states), row-stochastic.

 required

Returns:

Name Type Description
path ndarray

Most likely state sequence, shape (n_obs,).

resample_align_curve 

resample_align_curve(
    points: ndarray,
    original_timestep: float,
    target_timestep: float,
    align_length: int = -1,
) -> ndarray

Align a curve from one time grid to another via linear interpolation.

to_local_average_f0 

to_local_average_f0(
    hidden: Tensor,
    center: Tensor | None = None,
    thread: float = 0.03,
) -> ndarray

Decode hidden representation to F0 via local weighted average.

to_viterbi_f0 

to_viterbi_f0(
    hidden: Tensor, thread: float = 0.03
) -> ndarray

Decode hidden representation to F0 via Viterbi.

Uses a hand-written numpy Viterbi decoder (replacing librosa.sequence.viterbi) to avoid the librosa dependency.