HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement

Pavel Andreev, Aibek Alanov, Oleg Ivanov, Dmitry Vetrov

TL;DR

HiFi++ is a highly efficient, unified framework that adapts the popular HiFi-GAN vocoder architecture for speech bandwidth extension and speech enhancement. By introducing novel time- and frequency-domain processing modules, the model achieves state-of-the-art quality while being significantly more lightweight—sometimes by two orders of magnitude—than competing baselines.

What Problem Does This Paper Address?

In audio processing, two major challenges frequently arise due to poor recording equipment or restrictive transmission channels:

  1. Bandwidth Extension (BWE): Also known as audio super-resolution, this involves restoring high-frequency details to low-sampling-rate audio.
  2. Speech Enhancement (SE): This involves denoising an audio signal to remove extraneous background noise.

While Generative Adversarial Networks (GANs) have revolutionized neural vocoding (converting spectrograms back into waveforms), adapting them efficiently to BWE and SE has been tricky. Existing state-of-the-art models are either computationally massive or process audio strictly in one domain (e.g., only using spectrograms), which causes them to miss fine-grained waveform details and drop performance on classic relative audio metrics. The authors address this gap by building a lightweight, unified GAN framework that handles both time- and frequency-domain information seamlessly.

Methods

The core innovation of HiFi++ is its modified generator architecture, which appends three new modules to a standard HiFi-GAN (V2) upsampler core.

Fig. 1. HiFi++ architecture and training pipeline. The HiFi++ generator consists of the HiFi-like Upsampler and three intro- duced modules SpectralUNet, WaveUNet and SpectralMaskNet (their sizes are in yellow boxes). The generator’s architecture is identical for BWE and SE.

The pipeline operates as follows:

  • SpectralUNet: A 2D UNet-like convolutional network that preprocesses the input mel-spectrogram, correcting and extracting essential features before it is converted into a waveform.
  • Upsampler (HiFi-like): The baseline vocoder core that scales the preprocessed spectrogram up toward a waveform representation.
  • WaveUNet: A 1D time-domain post-processing module. It takes the upsampler's output concatenated with the original input waveform to correct errors directly in the time domain.
  • SpectralMaskNet: The final module, which performs frequency-domain post-processing. It applies a channel-wise Short-Time Fourier Transform (STFT), predicts multiplicative mask factors to clean up amplitudes without altering the phase, and applies an inverse STFT to output the final clean waveform.

For training, the authors utilize a multi-discriminator adversarial framework with a combination of Least Squares GAN (LS-GAN) loss, feature matching loss, and mel-spectrogram loss. To keep the model lightweight, they use scaled-down, identical discriminators with 4x fewer channels than traditional setups.

Datasets

The framework was evaluated on two primary datasets:

  • VCTK: Used for the BWE task. It contains 44,200 speech recordings from 110 speakers. The authors carefully split the data to prevent speaker- or text-level leakage. High-resolution targets (16 kHz) were downsampled to 1 kHz, 2 kHz, and 4 kHz for training and evaluation.
  • VCTK-DEMAND: Used for the SE (denoising) task. The training set includes 28 speakers mixed with noise at four Signal-to-Noise Ratios (SNRs: 0, 5, 10, and 15 dB), while the test set features different speakers and noise levels (2.5, 7.5, 12.5, and 17.5 dB).

Key Insights and Ideas

  • The Power of Dual-Domain Processing: Combining time-domain corrections (via WaveUNet) with frequency-domain masking (via SpectralMaskNet) allows the model to eliminate artifacts and noise far more effectively than operating in a single domain. Ablation studies proved both modules are absolutely essential to performance.
  • Adversarial Loss is Crucial for BWE: Purely supervised reconstruction models (like TFilm) struggle significantly at very low input bandwidths (e.g., 1 kHz). Adversarial objectives allow the model to generate plausible high-frequency details rather than just blurry averages.
  • Waveform Inputs Matter: Spectrogram-only models like VoiceFixer can achieve high subjective quality but perform poorly on relative metrics like SI-SDR and STOI because they completely discard the phase and raw structure of the input signal. Explicitly utilizing the raw waveform as an input prevents this information loss.

Key Results

HiFi++ delivers a remarkable trade-off between audio quality and computational efficiency:

  • Bandwidth Extension: HiFi++ outperformed the strongest baseline (SEANet) across all tested bandwidths (1 kHz, 2 kHz, and 4 kHz) under Mean Opinion Score (MOS) evaluations, despite being 5 times smaller (1.7M parameters vs. 9.2M).
  • Speech Enhancement: HiFi++ achieved comparable performance to state-of-the-art models like DB-AIAT and VoiceFixer while requiring drastically fewer resources. Specifically, HiFi++ uses just 1.7M parameters and 2.8G MACs, compared to DB-AIAT's 2.8M parameters / 41.8G MACs and VoiceFixer's massive 122.1M parameters / 34.4G MACs.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Generative adversarial networks have recently demonstrated outstanding performance in neural vocoding outperforming best autoregressive and flow-based models. In this paper, we show that this success can be extended to other tasks of conditional audio generation. In particular, building upon HiFi vocoders, we propose a novel HiFi++ general framework for bandwidth extension and speech enhancement. We show that with the improved generator architecture, HiFi++ performs better or comparably with the state-of-the-art in these tasks while spending significantly less computational resources. The effectiveness of our approach is validated through a series of extensive experiments.
BibTeX
@inproceedings{andreev2023hifi,
  author = {Pavel Andreev and Aibek Alanov and Oleg Ivanov and Dmitry Vetrov},
  title = {{HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=kizL9PHivX},
}
Figure