TL;DR
HiFi++ is a highly efficient, unified framework that adapts the popular HiFi-GAN vocoder architecture for speech bandwidth extension and speech enhancement. By introducing novel time- and frequency-domain processing modules, the model achieves state-of-the-art quality while being significantly more lightweight—sometimes by two orders of magnitude—than competing baselines.
What Problem Does This Paper Address?
In audio processing, two major challenges frequently arise due to poor recording equipment or restrictive transmission channels:
- Bandwidth Extension (BWE): Also known as audio super-resolution, this involves restoring high-frequency details to low-sampling-rate audio.
- Speech Enhancement (SE): This involves denoising an audio signal to remove extraneous background noise.
While Generative Adversarial Networks (GANs) have revolutionized neural vocoding (converting spectrograms back into waveforms), adapting them efficiently to BWE and SE has been tricky. Existing state-of-the-art models are either computationally massive or process audio strictly in one domain (e.g., only using spectrograms), which causes them to miss fine-grained waveform details and drop performance on classic relative audio metrics. The authors address this gap by building a lightweight, unified GAN framework that handles both time- and frequency-domain information seamlessly.
Methods
The core innovation of HiFi++ is its modified generator architecture, which appends three new modules to a standard HiFi-GAN (V2) upsampler core.
The pipeline operates as follows:
- SpectralUNet: A 2D UNet-like convolutional network that preprocesses the input mel-spectrogram, correcting and extracting essential features before it is converted into a waveform.
- Upsampler (HiFi-like): The baseline vocoder core that scales the preprocessed spectrogram up toward a waveform representation.
- WaveUNet: A 1D time-domain post-processing module. It takes the upsampler's output concatenated with the original input waveform to correct errors directly in the time domain.
- SpectralMaskNet: The final module, which performs frequency-domain post-processing. It applies a channel-wise Short-Time Fourier Transform (STFT), predicts multiplicative mask factors to clean up amplitudes without altering the phase, and applies an inverse STFT to output the final clean waveform.
For training, the authors utilize a multi-discriminator adversarial framework with a combination of Least Squares GAN (LS-GAN) loss, feature matching loss, and mel-spectrogram loss. To keep the model lightweight, they use scaled-down, identical discriminators with 4x fewer channels than traditional setups.
Datasets
The framework was evaluated on two primary datasets:
- VCTK: Used for the BWE task. It contains 44,200 speech recordings from 110 speakers. The authors carefully split the data to prevent speaker- or text-level leakage. High-resolution targets (16 kHz) were downsampled to 1 kHz, 2 kHz, and 4 kHz for training and evaluation.
- VCTK-DEMAND: Used for the SE (denoising) task. The training set includes 28 speakers mixed with noise at four Signal-to-Noise Ratios (SNRs: 0, 5, 10, and 15 dB), while the test set features different speakers and noise levels (2.5, 7.5, 12.5, and 17.5 dB).
Key Insights and Ideas
- The Power of Dual-Domain Processing: Combining time-domain corrections (via WaveUNet) with frequency-domain masking (via SpectralMaskNet) allows the model to eliminate artifacts and noise far more effectively than operating in a single domain. Ablation studies proved both modules are absolutely essential to performance.
- Adversarial Loss is Crucial for BWE: Purely supervised reconstruction models (like TFilm) struggle significantly at very low input bandwidths (e.g., 1 kHz). Adversarial objectives allow the model to generate plausible high-frequency details rather than just blurry averages.
- Waveform Inputs Matter: Spectrogram-only models like VoiceFixer can achieve high subjective quality but perform poorly on relative metrics like SI-SDR and STOI because they completely discard the phase and raw structure of the input signal. Explicitly utilizing the raw waveform as an input prevents this information loss.
Key Results
HiFi++ delivers a remarkable trade-off between audio quality and computational efficiency:
- Bandwidth Extension: HiFi++ outperformed the strongest baseline (SEANet) across all tested bandwidths (1 kHz, 2 kHz, and 4 kHz) under Mean Opinion Score (MOS) evaluations, despite being 5 times smaller (1.7M parameters vs. 9.2M).
- Speech Enhancement: HiFi++ achieved comparable performance to state-of-the-art models like DB-AIAT and VoiceFixer while requiring drastically fewer resources. Specifically, HiFi++ uses just 1.7M parameters and 2.8G MACs, compared to DB-AIAT's 2.8M parameters / 41.8G MACs and VoiceFixer's massive 122.1M parameters / 34.4G MACs.
Limitations and Future Work
Not discussed in the paper.
Summary updated