TL;DR
This paper introduces ScoreDec, a novel audio codec that significantly improves the quality of compressed audio by using a score-based diffusion post-filter. Unlike previous methods that rely on complex GAN training to achieve high fidelity, ScoreDec uses a more interpretable approach that preserves phase information, leading to more natural-sounding audio at a competitive 24 kbps bitrate.
What Problem Does This Paper Address?
Modern end-to-end (E2E) neural audio codecs can achieve impressive compression ratios, but they often struggle to fully bridge the quality gap with natural audio, especially at lower bitrates. A key challenge is accurately reconstructing the phase of the audio signal. Phase information is crucial for the perceived directionality and spatiality of sound. Traditional E2E codecs often use Generative Adversarial Networks (GANs) to improve reconstruction quality, but GANs can be tricky to train and, more importantly, tend to generate plausible but not necessarily original phase information. This leads to artifacts and a loss of naturalness, particularly in complex audio scenes. The paper aims to overcome these limitations by developing a codec that preserves original phase information without the need for GANs.
Methods
The core innovation is a Score-Based Diffusion Post-Filter (SPF) applied in the complex spectral domain. This SPF is integrated with a pre-existing neural audio codec called AudioDec. Here's a breakdown:
- AudioDec (Base Codec): This is a standard E2E neural codec that encodes audio into a compressed representation and then decodes it. It's trained in two stages: first with simple metric losses (like mel-spectrogram loss) for faster training, and then with adversarial losses (GANs) to refine the waveform.
- Score-Based Diffusion Post-Filter (SPF): This is the novel component. It takes the output of the AudioDec's first stage (which has good magnitude but potentially flawed phase) and refines it. The SPF is based on Score-Based Generative Models (SGMs), specifically adapted for speech enhancement. SGMs work by learning to reverse a diffusion process that gradually adds noise to data. In this context, the SPF learns to "denoise" the complex spectral representation of the audio, effectively restoring both magnitude and phase.
- Training: Crucially, the SPF is trained only using spectral and score-matching losses. This avoids the complexities and phase-obfuscating nature of GAN training. The entire ScoreDec system is trained by first training the AudioDec's encoder-decoder with metric losses and then training the SPF using the score-matching objective on the output of the first-stage decoder.
- Complex Spectral Domain: The SPF operates on the complex Short-Time Fourier Transform (STFT) of the audio. To handle the data representation, an amplitude companding and scaling step is applied to normalize the complex spectra before processing.
Datasets
- Training: The VCTK dataset, derived from 84 gender-balanced English speakers, was used for training.
- Testing: Two speakers (one female, one male) from the Valentini dataset were used for testing.
- Evaluation: The paper also evaluates the proposed SPF on the DSP-based Opus codec, demonstrating its generality.
Key Insights and Ideas
- Phase Preservation is Key: The paper strongly emphasizes that accurate phase reconstruction is critical for achieving human-level naturalness in audio codecs, and GANs are not the ideal tool for this.
- Score-Based Models for Restoration: Score-based diffusion models, known for their generative capabilities, can be effectively repurposed as powerful post-filters for audio enhancement, particularly for restoring phase information.
- Interpretable Training: By avoiding GANs and relying on metric and score-matching losses, ScoreDec offers a more interpretable and stable training process.
- Generality of SPF: The score-based diffusion post-filter is shown to be effective not only for neural codecs like AudioDec but also for improving existing DSP-based codecs like Opus.
Key Results
- Objective Evaluation: ScoreDec significantly outperforms its base codec (AudioDec) and even the standard Opus codec in terms of waveform similarity (measured by Wav and SI-SDR), indicating superior phase preservation. For example, SI-SDR improved from -0.50 dB for AudioDec to 8.17 dB for ScoreDec.
- Subjective Evaluation (MOS): ScoreDec achieves "human-level naturalness" at a 24 kbps bitrate, matching or exceeding the quality of natural speech in subjective listening tests. The SPF also boosts the quality of the Opus codec to human-level naturalness.
- Phase Reconstruction: Visual comparisons of spectrograms and phase plots (Figure 2 and 3) clearly show that ScoreDec preserves the original phase much better than AudioDec (even with GAN training) and Opus.
Limitations and Future Work
- Inference Speed: The paper acknowledges that ScoreDec is significantly slower than traditional codecs, with a real-time factor (RTF) of 1.707. This is attributed to its non-causal architecture and iterative inference process, making it unsuitable for streaming applications.
- Future Work: The authors suggest exploring causal architectures and fast inference techniques to enable streaming capabilities. They also propose evaluating ScoreDec's effectiveness for general spatial audio signals beyond speech.
Summary updated