Skip to content

vllm_omni.diffusion.models.soulx_singer.modules.vocoder

AdaLayerNorm

Bases: Module

Adaptive Layer Normalization module with learnable embeddings per num_embeddings classes

Parameters:

Name Type Description Default
num_embeddings int

Number of embeddings.

 required
embedding_dim int

Dimension of the embeddings.

 required

dim instance-attribute 

dim = embedding_dim

eps instance-attribute 

eps = eps

scale instance-attribute 

scale = nn.Embedding(
    num_embeddings=num_embeddings,
    embedding_dim=embedding_dim,
)

shift instance-attribute 

shift = nn.Embedding(
    num_embeddings=num_embeddings,
    embedding_dim=embedding_dim,
)

forward 

forward(x: Tensor, cond_embedding_id: Tensor) -> Tensor

Backbone

Bases: Module

Base class for the generator's backbone. It preserves the same temporal resolution across all layers.

forward 

forward(x: Tensor, **kwargs) -> Tensor

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (B, C, L), where B is the batch size, C denotes output features, and L is the sequence length.

 required

Returns:

Name Type Description
Tensor Tensor

Output of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension.

ConvNeXtBlock

Bases: Module

ConvNeXt Block adapted from https://github.com/facebookresearch/ConvNeXt to 1D audio signal.

Parameters:

Name Type Description Default
dim int

Number of input channels.

 required
intermediate_dim int

Dimensionality of the intermediate layer.

 required
layer_scale_init_value float

Initial value for the layer scale. None means no scaling. Defaults to None.

 required
adanorm_num_embeddings int

Number of embeddings for AdaLayerNorm. None means non-conditional LayerNorm. Defaults to None.

 None

act instance-attribute 

act = nn.GELU()

adanorm instance-attribute 

adanorm = adanorm_num_embeddings is not None

dwconv instance-attribute 

dwconv = nn.Conv1d(
    dim, dim, kernel_size=7, padding=3, groups=dim
)

gamma instance-attribute 

gamma = (
    nn.Parameter(
        layer_scale_init_value * torch.ones(dim),
        requires_grad=True,
    )
    if layer_scale_init_value > 0
    else None
)

norm instance-attribute 

norm = AdaLayerNorm(adanorm_num_embeddings, dim, eps=1e-06)

pwconv1 instance-attribute 

pwconv1 = nn.Linear(dim, intermediate_dim)

pwconv2 instance-attribute 

pwconv2 = nn.Linear(intermediate_dim, dim)

forward 

forward(
    x: Tensor, cond_embedding_id: Tensor | None = None
) -> Tensor

FourierHead

Bases: Module

Base class for inverse fourier modules.

forward 

forward(x: Tensor) -> Tensor

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension.

 required

Returns:

Name Type Description
Tensor Tensor

Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.

IMDCT

Bases: Module

Inverse Modified Discrete Cosine Transform (IMDCT) module.

Parameters:

Name Type Description Default
frame_len int

Length of the MDCT frame.

 required
padding str

Type of padding. Options are "center" or "same". Defaults to "same".

 'same'

frame_len instance-attribute 

frame_len = frame_len

padding instance-attribute 

padding = padding

forward 

forward(x: Tensor) -> Tensor

Apply the Inverse Modified Discrete Cosine Transform (IMDCT) to the input MDCT coefficients.

Parameters:

Name Type Description Default
x Tensor

Input MDCT coefficients of shape (B, L, N), where B is the batch size, L is the number of frames, and N is the number of frequency bins.

 required

Returns:

Name Type Description
Tensor Tensor

Reconstructed audio waveform of shape (B, T), where T is the length of the audio.

IMDCTCosHead

Bases: FourierHead

IMDCT Head module for predicting MDCT coefficients with parametrizing MDCT = exp(m) · cos(p)

Parameters:

Name Type Description Default
dim int

Hidden dimension of the model.

 required
mdct_frame_len int

Length of the MDCT frame.

 required
padding str

Type of padding. Options are "center" or "same". Defaults to "same".

 'same'
clip_audio bool

If True, clip audio to [-1.0, 1.0]. Defaults to False.

 False

clip_audio instance-attribute 

clip_audio = clip_audio

imdct instance-attribute 

imdct = IMDCT(frame_len=mdct_frame_len, padding=padding)

out instance-attribute 

out = nn.Linear(dim, mdct_frame_len)

forward 

forward(x: Tensor) -> Tensor

Forward pass of the IMDCTCosHead module.

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension.

 required

Returns:

Name Type Description
Tensor Tensor

Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.

IMDCTSymExpHead

Bases: FourierHead

IMDCT Head module for predicting MDCT coefficients with symmetric exponential function

Parameters:

Name Type Description Default
dim int

Hidden dimension of the model.

 required
mdct_frame_len int

Length of the MDCT frame.

 required
padding str

Type of padding. Options are "center" or "same". Defaults to "same".

 'same'
sample_rate int

The sample rate of the audio. If provided, the last layer will be initialized based on perceptual scaling. Defaults to None.

 None
clip_audio bool

If True, clip audio to [-1.0, 1.0]. Defaults to False.

 False

clip_audio instance-attribute 

clip_audio = clip_audio

imdct instance-attribute 

imdct = IMDCT(frame_len=mdct_frame_len, padding=padding)

out instance-attribute 

out = nn.Linear(dim, out_dim)

forward 

forward(x: Tensor) -> Tensor

Forward pass of the IMDCTSymExpHead module.

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension.

 required

Returns:

Name Type Description
Tensor Tensor

Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.

ISTFT

Bases: Module

Custom implementation of ISTFT since torch.istft doesn't allow custom padding (other than center=True) with windowing. This is because the NOLA (Nonzero Overlap Add) check fails at the edges. See issue: https://github.com/pytorch/pytorch/issues/62323 Specifically, in the context of neural vocoding we are interested in "same" padding analogous to CNNs. The NOLA constraint is met as we trim padded samples anyway.

Parameters:

Name Type Description Default
n_fft int

Size of Fourier transform.

 required
hop_length int

The distance between neighboring sliding window frames.

 required
win_length int

The size of window frame and STFT filter.

 required
padding str

Type of padding. Options are "center" or "same". Defaults to "same".

 'same'

hop_length instance-attribute 

hop_length = hop_length

n_fft instance-attribute 

n_fft = n_fft

padding instance-attribute 

padding = padding

win_length instance-attribute 

win_length = win_length

forward 

forward(spec: Tensor) -> Tensor

Compute the Inverse Short Time Fourier Transform (ISTFT) of a complex spectrogram.

Parameters:

Name Type Description Default
spec Tensor

Input complex spectrogram of shape (B, N, T), where B is the batch size, N is the number of frequency bins, and T is the number of time frames.

 required

Returns:

Name Type Description
Tensor Tensor

Reconstructed time-domain signal of shape (B, L), where L is the length of the output signal.

ISTFTHead

Bases: FourierHead

ISTFT Head module for predicting STFT complex coefficients.

Parameters:

Name Type Description Default
dim int

Hidden dimension of the model.

 required
n_fft int

Size of Fourier transform.

 required
hop_length int

The distance between neighboring sliding window frames, which should align with the resolution of the input features.

 required
padding str

Type of padding. Options are "center" or "same". Defaults to "same".

 'same'

istft instance-attribute 

istft = ISTFT(
    n_fft=n_fft,
    hop_length=hop_length,
    win_length=n_fft,
    padding=padding,
)

out instance-attribute 

out = torch.nn.Linear(dim, out_dim)

forward 

forward(x: Tensor) -> Tensor

Forward pass of the ISTFTHead module.

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension.

 required

Returns:

Name Type Description
Tensor Tensor

Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.

JsonHParams

items 

items()

keys 

keys()

values 

values()

MDCT

Bases: Module

Modified Discrete Cosine Transform (MDCT) module.

Parameters:

Name Type Description Default
frame_len int

Length of the MDCT frame.

 required
padding str

Type of padding. Options are "center" or "same". Defaults to "same".

 'same'

frame_len instance-attribute 

frame_len = frame_len

padding instance-attribute 

padding = padding

forward 

forward(audio: Tensor) -> Tensor

Apply the Modified Discrete Cosine Transform (MDCT) to the input audio.

Parameters:

Name Type Description Default
audio Tensor

Input audio waveform of shape (B, T), where B is the batch size and T is the length of the audio.

 required

Returns:

Name Type Description
Tensor Tensor

MDCT coefficients of shape (B, L, N), where L is the number of output frames and N is the number of frequency bins.

ResBlock1

Bases: Module

ResBlock adapted from HiFi-GAN V1 (https://github.com/jik876/hifi-gan) with dilated 1D convolutions, but without upsampling layers.

Parameters:

Name Type Description Default
dim int

Number of input channels.

 required
kernel_size int

Size of the convolutional kernel. Defaults to 3.

 3
dilation tuple[int]

Dilation factors for the dilated convolutions. Defaults to (1, 3, 5).

 (1, 3, 5)
lrelu_slope float

Negative slope of the LeakyReLU activation function. Defaults to 0.1.

 0.1
layer_scale_init_value float

Initial value for the layer scale. None means no scaling. Defaults to None.

 None

convs1 instance-attribute 

convs1 = nn.ModuleList(
    [
        weight_norm(
            nn.Conv1d(
                dim,
                dim,
                kernel_size,
                1,
                dilation=dilation[0],
                padding=self.get_padding(
                    kernel_size, dilation[0]
                ),
            )
        ),
        weight_norm(
            nn.Conv1d(
                dim,
                dim,
                kernel_size,
                1,
                dilation=dilation[1],
                padding=self.get_padding(
                    kernel_size, dilation[1]
                ),
            )
        ),
        weight_norm(
            nn.Conv1d(
                dim,
                dim,
                kernel_size,
                1,
                dilation=dilation[2],
                padding=self.get_padding(
                    kernel_size, dilation[2]
                ),
            )
        ),
    ]
)

convs2 instance-attribute 

convs2 = nn.ModuleList(
    [
        weight_norm(
            nn.Conv1d(
                dim,
                dim,
                kernel_size,
                1,
                dilation=1,
                padding=self.get_padding(kernel_size, 1),
            )
        ),
        weight_norm(
            nn.Conv1d(
                dim,
                dim,
                kernel_size,
                1,
                dilation=1,
                padding=self.get_padding(kernel_size, 1),
            )
        ),
        weight_norm(
            nn.Conv1d(
                dim,
                dim,
                kernel_size,
                1,
                dilation=1,
                padding=self.get_padding(kernel_size, 1),
            )
        ),
    ]
)

gamma instance-attribute 

gamma = nn.ParameterList(
    [
        nn.Parameter(
            layer_scale_init_value * torch.ones(dim, 1),
            requires_grad=True,
        )
        if layer_scale_init_value is not None
        else None,
        nn.Parameter(
            layer_scale_init_value * torch.ones(dim, 1),
            requires_grad=True,
        )
        if layer_scale_init_value is not None
        else None,
        nn.Parameter(
            layer_scale_init_value * torch.ones(dim, 1),
            requires_grad=True,
        )
        if layer_scale_init_value is not None
        else None,
    ]
)

lrelu_slope instance-attribute 

lrelu_slope = lrelu_slope

forward 

forward(x: Tensor) -> Tensor

get_padding staticmethod 

get_padding(kernel_size: int, dilation: int = 1) -> int

remove_weight_norm 

remove_weight_norm()

STFT

Bases: Module

center instance-attribute 

center = center

hop_length instance-attribute 

hop_length = hop_length

n_fft instance-attribute 

n_fft = n_fft

win_length instance-attribute 

win_length = win_length

forward 

forward(x: Tensor) -> Tensor

Vocoder

Bases: Module

model instance-attribute 

model = load_vocos_model(
    ckpt_path=ckpt_path, config=model_cfg
)

forward 

forward(x)

Vocos

Bases: Module

backbone instance-attribute 

backbone = VocosBackbone(
    input_channels=input_channels,
    dim=dim,
    intermediate_dim=intermediate_dim,
    num_layers=num_layers,
    adanorm_num_embeddings=adanorm_num_embeddings,
)

head instance-attribute 

head = ISTFTHead(dim, n_fft, hop_size, padding)

forward 

forward(x)

VocosBackbone

Bases: Backbone

Vocos backbone module built with ConvNeXt blocks. Supports additional conditioning with Adaptive Layer Normalization

Parameters:

Name Type Description Default
input_channels int

Number of input features channels.

 required
dim int

Hidden dimension of the model.

 required
intermediate_dim int

Intermediate dimension used in ConvNeXtBlock.

 required
num_layers int

Number of ConvNeXtBlock layers.

 required
layer_scale_init_value float

Initial value for layer scaling. Defaults to 1 / num_layers.

 None
adanorm_num_embeddings int

Number of embeddings for AdaLayerNorm. None means non-conditional model. Defaults to None.

 None

adanorm instance-attribute 

adanorm = adanorm_num_embeddings is not None

convnext instance-attribute 

convnext = nn.ModuleList(
    [
        (
            ConvNeXtBlock(
                dim=dim,
                intermediate_dim=intermediate_dim,
                layer_scale_init_value=layer_scale_init_value,
                adanorm_num_embeddings=adanorm_num_embeddings,
            )
        )
        for _ in (range(num_layers))
    ]
)

embed instance-attribute 

embed = nn.Conv1d(
    input_channels, dim, kernel_size=7, padding=3
)

final_layer_norm instance-attribute 

final_layer_norm = nn.LayerNorm(dim, eps=1e-06)

input_channels instance-attribute 

input_channels = input_channels

norm instance-attribute 

norm = AdaLayerNorm(adanorm_num_embeddings, dim, eps=1e-06)

forward 

forward(x: Tensor, **kwargs) -> Tensor

VocosResNetBackbone

Bases: Backbone

Vocos backbone module built with ResBlocks.

Parameters:

Name Type Description Default
input_channels int

Number of input features channels.

 required
dim int

Hidden dimension of the model.

 required
num_blocks int

Number of ResBlock1 blocks.

 required
layer_scale_init_value float

Initial value for layer scaling. Defaults to None.

 None

embed instance-attribute 

embed = weight_norm(
    nn.Conv1d(input_channels, dim, kernel_size=3, padding=1)
)

input_channels instance-attribute 

input_channels = input_channels

resnet instance-attribute 

resnet = nn.Sequential(
    *[
        (
            ResBlock1(
                dim=dim,
                layer_scale_init_value=layer_scale_init_value,
            )
        )
        for _ in (range(num_blocks))
    ]
)

forward 

forward(x: Tensor, **kwargs) -> Tensor

build_vocoder_model 

build_vocoder_model(cfg)

load_checkpoint 

load_checkpoint(build_model_func, cfg, ckpt_path)

load_vocos_model 

load_vocos_model(
    ckpt_path: str | None = None,
    config: DictConfig = None,
    device: str = "cuda",
)

Load Vocos vocoder.

Parameters:

Name Type Description Default
config DictConfig

DictConfig, config for vocoder.

 None
ckpt_path str | None

str | None = None, path to checkpoint.

 None
device str

str, device to load model. Note: accelerate dispatch handles device placement.

 'cuda'

safe_log 

safe_log(x: Tensor, clip_val: float = 1e-07) -> Tensor

Computes the element-wise logarithm of the input tensor with clipping to avoid near-zero values.

Parameters:

Name Type Description Default
x Tensor

Input tensor.

 required
clip_val float

Minimum value to clip the input tensor. Defaults to 1e-7.

 1e-07

Returns:

Name Type Description
Tensor Tensor

Element-wise logarithm of the input tensor with clipping applied.

symexp 

symexp(x: Tensor) -> Tensor

symlog 

symlog(x: Tensor) -> Tensor