Abstract
Generative adversarial networks have recently demonstrated outstanding performance in neural vocoding outperforming best autoregressive and flow-based models. In this paper, we show that this success can be extended to other tasks of conditional audio generation. In particular, building upon HiFi vocoders, we propose a novel HiFi++ general framework for neural vocoding, bandwidth extension, and speech enhancement. We show that with the improved generator architecture and simplified multi-discriminator training, HiFi++ performs on par with the state-of-the-art in these tasks while spending significantly less memory and computational resources. The effectiveness of our approach is validated through a series of extensive experiments.
Introduction
In audio processing, two major challenges are Bandwidth Extension (BWE)—restoring high-frequency content to low-bitrate or "muffled" recordings—and Speech Enhancement (SE)—removing background noise from a signal. While Generative Adversarial Networks (GANs) like HiFi-GAN have revolutionized vocoding (converting mel-spectrograms to waveforms), applying them to BWE and SE has traditionally involved a trade-off between computational efficiency and audio quality. HiFi++ addresses this by providing a unified, lightweight architecture that handles these tasks with high fidelity.
The HiFi++ Framework
The HiFi++ generator is built upon the HiFi-GAN (V2) foundation, incorporating three specialized modules to handle spectral and temporal complexities:
- SpectralUNet: A 2D convolutional U-Net pre-processor that "cleans" the input mel-spectrogram before upsampling.
- WaveUNet: A 1D convolutional U-Net operating in the time domain. It corrects the waveform post-upsampling and merges the prediction with the original source signal for consistency.
- SpectralMaskNet: A frequency-domain post-processing step using learnable spectral masking to suppress residual noise without altering phase.
The training process utilizes a multi-discriminator setup optimized for efficiency, reducing discriminator parameters by 75% compared to standard implementations.
Hybrid Domain Processing
A core insight of HiFi++ is that high-quality speech restoration requires simultaneous processing in both time and frequency domains. The use of both 1D and 2D U-Net architectures allows the model to address different types of artifacts.
By combining WaveUNet (time domain) and SpectralMaskNet (frequency domain), the model corrects structural rhythmic issues while eliminating fine-grained spectral "hiss." This hybrid approach enables state-of-the-art results with only 1.7 million parameters, making it viable for real-time use on consumer hardware.
Experimental Evaluation
The framework was evaluated on standard benchmarks:
- Bandwidth Extension: Using the VCTK dataset, HiFi++ upscaled audio from 1kHz, 2kHz, and 4kHz to 16kHz. It outperformed the previous SOTA (SEANet) while being 5x smaller, and was nearly 70x smaller than VoiceFixer with better subjective quality.
- Speech Enhancement: On the VCTK-DEMAND dataset, the model achieved performance comparable to heavy models like DB-AIAT but required only 2.8 GFLOPs (vs. 41.8 GFLOPs for DB-AIAT).
The authors also introduced WV-MOS, an objective metric based on a fine-tuned wav2vec 2.0 model. They argue that traditional metrics like PESQ often fail to reward "natural" sounding speech from generative models, whereas WV-MOS correlates better with human perception.
Conclusion and Future Work
HiFi++ demonstrates that high-fidelity audio restoration does not require massive parameter counts. While currently focused on 16kHz sampling rates, future work includes scaling to super-wideband (32kHz) and full-band (48kHz) audio. Additionally, the researchers aim to further refine neural objective metrics like WV-MOS to better automate the evaluation of generative audio models, reducing reliance on expensive human listening tests.
Summary updated