HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement

Pavel Andreev, Aibek Alanov, Oleg Ivanov, Dmitry Vetrov

Abstract

Generative adversarial networks have recently demonstrated outstanding performance in neural vocoding outperforming best autoregressive and flow-based models. In this paper, we show that this success can be extended to other tasks of conditional audio generation. In particular, building upon HiFi vocoders, we propose a novel HiFi++ general framework for neural vocoding, bandwidth extension, and speech enhancement. We show that with the improved generator architecture and simplified multi-discriminator training, HiFi++ performs on par with the state-of-the-art in these tasks while spending significantly less memory and computational resources. The effectiveness of our approach is validated through a series of extensive experiments.

Introduction

In audio processing, two major challenges are Bandwidth Extension (BWE)—restoring high-frequency content to low-bitrate or "muffled" recordings—and Speech Enhancement (SE)—removing background noise from a signal. While Generative Adversarial Networks (GANs) like HiFi-GAN have revolutionized vocoding (converting mel-spectrograms to waveforms), applying them to BWE and SE has traditionally involved a trade-off between computational efficiency and audio quality. HiFi++ addresses this by providing a unified, lightweight architecture that handles these tasks with high fidelity.

The HiFi++ Framework

The HiFi++ generator is built upon the HiFi-GAN (V2) foundation, incorporating three specialized modules to handle spectral and temporal complexities:

Overview of the HiFi++ generator architecture and training pipeline.
  1. SpectralUNet: A 2D convolutional U-Net pre-processor that "cleans" the input mel-spectrogram before upsampling.
  2. WaveUNet: A 1D convolutional U-Net operating in the time domain. It corrects the waveform post-upsampling and merges the prediction with the original source signal for consistency.
  3. SpectralMaskNet: A frequency-domain post-processing step using learnable spectral masking to suppress residual noise without altering phase.

The training process utilizes a multi-discriminator setup optimized for efficiency, reducing discriminator parameters by 75% compared to standard implementations.

Hybrid Domain Processing

A core insight of HiFi++ is that high-quality speech restoration requires simultaneous processing in both time and frequency domains. The use of both 1D and 2D U-Net architectures allows the model to address different types of artifacts.

Detailed architectures of the 1D WaveUNet (left) and 2D SpectralUNet (right) modules.

By combining WaveUNet (time domain) and SpectralMaskNet (frequency domain), the model corrects structural rhythmic issues while eliminating fine-grained spectral "hiss." This hybrid approach enables state-of-the-art results with only 1.7 million parameters, making it viable for real-time use on consumer hardware.

Experimental Evaluation

The framework was evaluated on standard benchmarks:

  • Bandwidth Extension: Using the VCTK dataset, HiFi++ upscaled audio from 1kHz, 2kHz, and 4kHz to 16kHz. It outperformed the previous SOTA (SEANet) while being 5x smaller, and was nearly 70x smaller than VoiceFixer with better subjective quality.
  • Speech Enhancement: On the VCTK-DEMAND dataset, the model achieved performance comparable to heavy models like DB-AIAT but required only 2.8 GFLOPs (vs. 41.8 GFLOPs for DB-AIAT).

The authors also introduced WV-MOS, an objective metric based on a fine-tuned wav2vec 2.0 model. They argue that traditional metrics like PESQ often fail to reward "natural" sounding speech from generative models, whereas WV-MOS correlates better with human perception.

Conclusion and Future Work

HiFi++ demonstrates that high-fidelity audio restoration does not require massive parameter counts. While currently focused on 16kHz sampling rates, future work includes scaling to super-wideband (32kHz) and full-band (48kHz) audio. Additionally, the researchers aim to further refine neural objective metrics like WV-MOS to better automate the evaluation of generative audio models, reducing reliance on expensive human listening tests.

Summary updated

Abstract

Generative adversarial networks have recently demonstrated outstanding performance in neural vocoding outperforming best autoregressive and flow-based models. In this paper, we show that this success can be extended to other tasks of conditional audio generation. In particular, building upon HiFi vocoders, we propose a novel HiFi++ general framework for neural vocoding, bandwidth extension, and speech enhancement. We show that with the improved generator architecture and simplified multi-discriminator training, HiFi++ performs on par with the state-of-the-art in these tasks while spending significantly less memory and computational resources. The effectiveness of our approach is validated through a series of extensive experiments.
BibTeX
@inproceedings{andreev2023hifi,
  author = {Pavel Andreev and Aibek Alanov and Oleg Ivanov and Dmitry Vetrov},
  title = {{HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=kizL9PHivX},
}
Figure