SuperCodec: A Neural Speech Codec with Selective Back-Projection Network

Youqiang Zheng, Weiping Tu, Li Xiao, Xinmeng Xu

Introduction

Digital communication relies on efficient speech coding to achieve high-quality audio at minimal bitrates. While neural speech codecs have significantly advanced beyond traditional methods like Opus, they often encounter a "quality ceiling" when bitrates drop below 1.5 kbps. SuperCodec addresses this limitation by incorporating "back-projection" techniques, originally developed for image super-resolution, into a neural speech codec framework. This approach allows for high-fidelity speech reconstruction at ultra-low bitrates, outperforming existing state-of-the-art models.

Problem Statement

Most modern neural codecs utilize a standard encoder-decoder architecture. The encoder down-samples audio into a compact latent representation, which is then quantized and sent to the decoder for up-sampling. However, this standard approach suffers from two primary flaws:

  1. Information Loss: Simple convolutional down-sampling often discards non-linear information and fine acoustic details essential for realistic reconstruction.
  2. Reconstruction Challenges: Standard up-sampling methods, such as transposed convolutions, struggle to accurately recover missing high-resolution details from highly compressed data.

At bitrates below 1.5 kbps, these issues typically result in muffled or robotic-sounding speech.

Methodology

The core innovation of SuperCodec is the replacement of standard sampling layers with Selective Back-Projection modules.

SuperCodec Architecture: The pipeline includes an encoder with SDBP modules, a Quantizer, and a decoder with SUBP modules.

Selective Back-Projection (SDBP & SUBP)

Unlike one-way transformations, these modules implement a feedback loop to refine features:

  • Selective Down-sampling Back-Projection (SDBP): During encoding, the module projects low-resolution features back to a high-resolution space to calculate a residual error, which is then used to refine the final down-sampled output.
  • Selective Up-sampling Back-Projection (SUBP): During decoding, the module iteratively refines high-resolution features to ensure they remain faithful to the compressed input.

Selective Feature Fusion

To enhance these feedback loops, a Selective Feature Fusion (S) block is integrated. This block employs a self-attention mechanism—utilizing Global Average Pooling and Softmax—to adaptively weight feature maps, directing the network to focus on critical frequencies and details for natural-sounding speech.

Experimental Setup

The model was evaluated using the VCTK dataset, comprising 40 hours of audio from 100 speakers, down-sampled to 16 kHz. To test generalization, 8 speakers (4 male, 4 female) were excluded from training and used as "unseen" test subjects.

Results and Discussion

SuperCodec demonstrates significant improvements in subjective audio quality, as measured by MUSHRA scores.

MUSHRA scores vs. Bitrate: SuperCodec (Ours) outperforms Lyra-v2 and Encodec at significantly lower bitrates.

Key findings include:

  • Efficiency: SuperCodec at 1 kbps achieved higher quality scores than Lyra V2 at 3.2 kbps and Encodec at 6 kbps. At 3 kbps, it outperformed Encodec's 6 kbps mode, effectively doubling compression efficiency.
  • Ablation Insights: Studies revealed that the SUBP module in the decoder is more critical for quality than the encoder's SDBP, suggesting that reconstruction is the primary bottleneck in low-bitrate codecs.
  • Computational Performance: SuperCodec maintains a small footprint (14.66M parameters) and achieves a Real-Time Factor (RTF) of ~0.03 on a standard CPU, making it suitable for real-time applications.

Conclusion

SuperCodec successfully shatters the quality ceiling for ultra-low bitrate speech coding. By treating sampling as an iterative optimization problem through back-projection, it preserves fine acoustic textures that standard codecs lose. Future work may extend this approach to full-band audio (44.1/48 kHz) and explore integration with generative models to further reduce bitrates.

Summary updated

Abstract

Neural speech coding is a rapidly developing topic, where state-of-the-art approaches now exhibit superior compression performance than conventional methods. Despite significant progress, existing methods still have limitations in preserving and reconstructing fine details for optimal reconstruction, especially at low bitrates. In this study, we introduce SuperCodec, a neural speech codec that achieves state-of-the-art performance at low bitrates. It employs a novel back projection method with selective feature fusion for augmented representation. Specifically, we propose to use Selective Up-sampling Back Projection (SUBP) and Selective Down-sampling Back Projection (SDBP) modules to replace the standard up- and down-sampling layers at the encoder and decoder, respectively. Experimental results show that our method outperforms the existing neural speech codecs operating at various bitrates. Specifically, our proposed method can achieve higher quality reconstructed speech at 1 kbps than Lyra V2 at 3.2 kbps and Encodec at 6 kbps.
BibTeX
@inproceedings{zheng2024supercodec,
  author = {Youqiang Zheng and Weiping Tu and Li Xiao and Xinmeng Xu},
  title = {{SuperCodec: A Neural Speech Codec with Selective Back-Projection Network}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=7YyQFv4RDr},
}
Figure