Is GAN Necessary for Mel-Spectrogram-based Neural Vocoder?

Hui-Peng Du, Yang Ai, Rui-Chen Zheng, Ye-Xin Lu, Zhen-Hua Ling

TL;DR

FreeGAN demonstrates that high-fidelity speech synthesis doesn't strictly require the complexity of Generative Adversarial Networks (GANs). By combining a novel amplitude-prior input, an improved SNAKE-ConvNeXt v2 backbone, and a frequency-weighted phase loss, FreeGAN matches the audio quality of state-of-the-art GAN vocoders while being significantly faster to train and more efficient to deploy.

What Problem Does This Paper Address?

In the world of speech synthesis, "vocoders" are the engines that turn abstract representations (like mel-spectrograms) into actual audio waveforms. For years, the gold standard has been GAN-based models like HiFi-GAN and BigVGAN. While they produce excellent audio, they come with heavy baggage:

  1. Training Complexity: GANs are notoriously finicky to train due to the adversarial dance between the generator and discriminator.
  2. Resource Overhead: Discriminators often contain 5x more parameters than the generator itself. While not used during inference, they must be stored for fine-tuning, bloating storage requirements.
  3. Efficiency: High-fidelity GANs can be computationally expensive, making real-time CPU generation difficult.

The authors ask a fundamental question: Can we achieve GAN-level quality without using a GAN at all?

Methods

The authors propose FreeGAN, a vocoder that is "free" from adversarial training. It uses an amplitude-phase serial prediction framework followed by an inverse Short-Time Fourier Transform (iSTFT) to reconstruct the waveform. To compensate for the lack of a GAN discriminator, they introduce three key innovations:

  • Input Design (Amplitude Prior): Instead of feeding the raw mel-spectrogram directly into the network, they use a "Mel pseudo-inverse filter" to create an initial amplitude estimate. This "prior" simplifies the model's job, allowing it to focus on refining details rather than learning the basic structure from scratch.
  • Architecture (SNAKE-ConvNeXt v2): They adapt the modern ConvNeXt v2 backbone but replace the standard GELU activation with the SNAKE activation. SNAKE is specifically designed to model periodic signals (like the oscillations in speech), significantly boosting the model's ability to predict accurate phase and amplitude.
  • Loss Function (FWAW Phase Loss): Without a GAN, high-frequency phase information often degrades. The authors designed a Frequency-Weighted Anti-Wrapping (FWAW) phase loss. This loss assigns higher weights to specific frequency bins, forcing the model to pay closer attention to the phase components that are hardest to reconstruct.

Datasets

The models were trained and evaluated on the VCTK-0.92 dataset, which contains approximately 44 hours of speech from 108 English speakers. The audio was downsampled to 16 kHz for the experiments. The authors used a standard split of 100 speakers for training/validation and 8 unseen speakers for testing.

Key Insights and Ideas

  • The "Phase Gap": GAN discriminators are effective because they implicitly provide the phase information that mel-spectrograms lack. FreeGAN proves that if you provide explicit, high-quality phase supervision (via FWAW loss) and a strong architecture, you don't need the adversarial signal.
  • Input Pre-processing is Underrated: The "Amplitude Prior" was found to be the most critical component in the ablation studies. By giving the model a "head start" with a pseudo-inverse filter, the overall Mean Cepstral Distortion (MCD) improved drastically.
  • Universal Applicability: These strategies aren't just for FreeGAN. The authors showed that adding their improvements to other non-GAN vocoders (like APNet2 and FreeV) allowed them to reach performance levels nearly identical to their GAN-trained versions.

Key Results

  • Quality: In Mean Opinion Score (MOS) tests, FreeGAN achieved a score of 4.12, statistically comparable to the heavy-hitting BigVGAN (4.14) and APNet2 (4.13).
  • Training Speed: FreeGAN is a speed demon. It trains roughly 15 times faster than BigVGAN (191 s/epoch vs. 1937 s/epoch).
  • Efficiency: It requires only 2.70 GFLOPS to generate 1 second of speech, making it one of the most lightweight high-quality options available.
  • Storage: Because there is no discriminator, the storage requirement is just the 13.4M parameters of the generator, whereas GAN models require an additional 40M to 70M parameters for their discriminators during the training/fine-tuning phase.

Limitations and Future Work

While FreeGAN performs exceptionally well at 16 kHz, the paper does not extensively explore performance at higher sampling rates (like 44.1 kHz or 48 kHz), where phase reconstruction becomes even more challenging. The authors state that their future work will involve integrating FreeGAN into practical downstream tasks like end-to-end text-to-speech (TTS) and low-bitrate speech coding to see how it holds up in production environments.

Summary updated

Abstract

Recently, mainstream mel-spectrogram-based neural vocoders rely on generative adversarial network (GAN) for high-fidelity speech generation, e.g., HiFi-GAN and BigVGAN. However, the use of GAN restricts training efficiency and model complexity. Therefore, this paper proposes a novel FreeGAN vocoder, aiming to answer the question of whether GAN is necessary for mel-spectrogram-based neural vocoders. The FreeGAN employs an amplitude-phase serial prediction framework, eliminating the need for GAN training. It incorporates amplitude prior input, SNAKE-ConvNeXt v2 backbone and frequency-weighted anti-wrapping phase loss to compensate for the performance loss caused by the absence of GAN. Experimental results confirm that the speech quality of FreeGAN is comparable to that of advanced GAN-based vocoders, while significantly improving training efficiency and complexity. Other explicit-phase-prediction-based neural vocoders can also work without GAN, leveraging our proposed methods.
BibTeX
@article{du2025is,
  author = {Hui-Peng Du and Yang Ai and Rui-Chen Zheng and Ye-Xin Lu and Zhen-Hua Ling},
  title = {{Is GAN Necessary for Mel-Spectrogram-based Neural Vocoder?}},
  journal = {IEEE Signal Processing Letters},
  year = {2025},
  doi = {10.1109/lsp.2025.3598729},
}

Loading related content...

Figure