Introduction
Digital communication relies on efficient speech coding to achieve high-quality audio at minimal bitrates. While neural speech codecs have significantly advanced beyond traditional methods like Opus, they often encounter a "quality ceiling" when bitrates drop below 1.5 kbps. SuperCodec addresses this limitation by incorporating "back-projection" techniques, originally developed for image super-resolution, into a neural speech codec framework. This approach allows for high-fidelity speech reconstruction at ultra-low bitrates, outperforming existing state-of-the-art models.
Problem Statement
Most modern neural codecs utilize a standard encoder-decoder architecture. The encoder down-samples audio into a compact latent representation, which is then quantized and sent to the decoder for up-sampling. However, this standard approach suffers from two primary flaws:
- Information Loss: Simple convolutional down-sampling often discards non-linear information and fine acoustic details essential for realistic reconstruction.
- Reconstruction Challenges: Standard up-sampling methods, such as transposed convolutions, struggle to accurately recover missing high-resolution details from highly compressed data.
At bitrates below 1.5 kbps, these issues typically result in muffled or robotic-sounding speech.
Methodology
The core innovation of SuperCodec is the replacement of standard sampling layers with Selective Back-Projection modules.
Selective Back-Projection (SDBP & SUBP)
Unlike one-way transformations, these modules implement a feedback loop to refine features:
- Selective Down-sampling Back-Projection (SDBP): During encoding, the module projects low-resolution features back to a high-resolution space to calculate a residual error, which is then used to refine the final down-sampled output.
- Selective Up-sampling Back-Projection (SUBP): During decoding, the module iteratively refines high-resolution features to ensure they remain faithful to the compressed input.
Selective Feature Fusion
To enhance these feedback loops, a Selective Feature Fusion (S) block is integrated. This block employs a self-attention mechanism—utilizing Global Average Pooling and Softmax—to adaptively weight feature maps, directing the network to focus on critical frequencies and details for natural-sounding speech.
Experimental Setup
The model was evaluated using the VCTK dataset, comprising 40 hours of audio from 100 speakers, down-sampled to 16 kHz. To test generalization, 8 speakers (4 male, 4 female) were excluded from training and used as "unseen" test subjects.
Results and Discussion
SuperCodec demonstrates significant improvements in subjective audio quality, as measured by MUSHRA scores.
Key findings include:
- Efficiency: SuperCodec at 1 kbps achieved higher quality scores than Lyra V2 at 3.2 kbps and Encodec at 6 kbps. At 3 kbps, it outperformed Encodec's 6 kbps mode, effectively doubling compression efficiency.
- Ablation Insights: Studies revealed that the SUBP module in the decoder is more critical for quality than the encoder's SDBP, suggesting that reconstruction is the primary bottleneck in low-bitrate codecs.
- Computational Performance: SuperCodec maintains a small footprint (14.66M parameters) and achieves a Real-Time Factor (RTF) of ~0.03 on a standard CPU, making it suitable for real-time applications.
Conclusion
SuperCodec successfully shatters the quality ceiling for ultra-low bitrate speech coding. By treating sampling as an iterative optimization problem through back-projection, it preserves fine acoustic textures that standard codecs lose. Future work may extend this approach to full-band audio (44.1/48 kHz) and explore integration with generative models to further reduce bitrates.
Summary updated