Abstract
APCodec is a novel neural audio codec designed to deliver high-fidelity 48 kHz audio at ultra-low bitrates, specifically targeting 6 kbps. Unlike traditional neural codecs that process raw waveforms, APCodec utilizes a parallel encoding and decoding architecture that separately handles amplitude and phase spectra. This approach significantly reduces computational complexity while maintaining superior sound quality. Experimental results demonstrate that APCodec outperforms industry standards such as Encodec and DAC in both subjective quality and inference speed, achieving up to 89× real-time performance on GPU.
Introduction
The development of digital audio codecs is governed by a persistent trilemma: balancing bitrate, reconstruction quality, and latency. While traditional parametric codecs are highly efficient, they often suffer from "robotic" artifacts. Conversely, modern neural waveform codecs like Encodec and SoundStream offer high fidelity but at a high computational cost. These models typically require extensive downsampling and upsampling operations to compress raw waveforms into low-bitrate representations, leading to increased complexity and latency.
APCodec (Amplitude and Phase Codec) addresses these limitations by shifting the processing domain from the time domain (waveforms) to the frequency domain (spectra). By explicitly modeling the amplitude and phase components in parallel, APCodec achieves high-fidelity 48 kHz audio reconstruction at a mere 6 kbps, bridging the gap between parametric efficiency and neural generative quality.
Methodology
Parallel Spectral Processing
The core innovation of APCodec lies in its input representation. Instead of raw audio samples, the encoder receives log amplitude and phase spectra extracted via Short-Time Fourier Transform (STFT). This spectral approach allows the model to bypass the deep, complex downsampling stacks required by waveform-based models. While a model like Encodec might require a 320× downsampling ratio to reach its target bitrate, APCodec only requires an 8× ratio because the input is already a condensed spectral representation.
Architecture and Quantization
The backbone of APCodec is based on a modified ConvNeXt v2 network, optimized for spectral data. To compress the continuous latent code into discrete tokens suitable for transmission, the system employs Residual Vector Quantization (RVQ). This allows for a flexible bitrate while maintaining high information density.
Hybrid Training and Distillation
To ensure natural-sounding output, APCodec is trained using a multi-faceted loss function:
- Spectral-level losses: To ensure the global "shape" and energy distribution of the sound are accurate.
- Generative Adversarial Network (GAN) losses: To synthesize fine-grained details and textures that human listeners perceive as high fidelity.
For real-time applications, the authors introduced APCodec-S, a streamable "student" model. This version utilizes causal convolutions and feed-forward layers to achieve a fixed latency of 6.67 ms. To prevent quality degradation in this faster version, the authors employed Knowledge Distillation, where the high-performing non-causal "teacher" model guides the training of the student, helping it capture complex phase relationships even with a limited temporal view.
Experimental Setup
The researchers evaluated APCodec across a diverse range of audio data to ensure robustness and generalization:
- VCTK & Common Voice: Large-scale speech corpora for multi-speaker evaluation.
- Opencpop: High-quality Mandarin singing voices to test melodic accuracy.
- FSD50K: Diverse environmental sound events (e.g., footsteps, breaking glass) to test non-speech performance.
Key Results and Discussion
Superior Quality at Low Bitrates
In MUSHRA (MUlti-Stimulus test with Hidden Reference and Anchor) subjective listening tests, APCodec consistently outperformed Encodec, AudioDec, and DAC at the 6 kbps bitrate. The explicit modeling of phase proved to be the "secret sauce," allowing the codec to maintain clarity and temporal alignment that other models often lose at such high compression ratios.
Computational Efficiency
APCodec demonstrates significant gains in inference speed:
- GPU Performance: 89× real-time generation.
- CPU Performance: 5.8× real-time generation, which is approximately 14 times faster than the Descript Audio Codec (DAC).
Generalization
Beyond speech, APCodec showed stronger generalization on singing and environmental noises compared to DAC. This suggests that the spectral domain approach is more versatile than direct waveform modeling, as it captures the fundamental frequency components of any sound type more efficiently.
Conclusion and Future Work
APCodec represents a significant step forward in neural audio compression, proving that frequency-domain modeling can yield both higher quality and better efficiency than time-domain approaches. While the streamable APCodec-S model shows a slight quality gap compared to its non-causal counterpart, it remains a highly viable solution for low-latency communication.
Future research will explore alternative spectral domains, such as the Modified Discrete Cosine Transform (MDCT), and investigate the utility of APCodec tokens for downstream tasks like zero-shot Text-to-Speech (TTS) and speech enhancement.
Summary updated