Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition

Krishna C. Puvvada, Nithin Rao Koluguri, Kunal Dhawan, Jagadeesh Balam, Boris Ginsburg

Introduction

Traditional audio processing for speech and speaker recognition relies heavily on continuous representations like Mel-spectrograms. However, the rise of foundational audio models inspired by Large Language Models (LLMs) has sparked interest in discrete "tokenized" audio. This paper investigates whether discrete audio tokens from neural codecs, specifically EnCodec, can serve as a viable, compressed alternative to Mel-spectrograms for discriminative tasks.

Methodology

The researchers evaluated EnCodec, which utilizes Residual Vector Quantization (RVQ) to approximate audio signals through a hierarchical series of codebooks. These tokens were integrated into two primary architectures: TitaNet for speaker verification and diarization, and Fast Conformer for automatic speech recognition (ASR). The study compared these discrete representations against standard Mel-spectrogram baselines across diverse datasets, including VoxCeleb, NIST-SRE, and LibriSpeech.

Key Insights: The Low-Pass Effect and Robustness

A pivotal finding of the study is that the EnCodec tokenizer inherently acts as a low-pass filter, significantly attenuating frequencies above 6kHz.

Estimated transfer function of the EnCodec tokenizer showing attenuation above 6kHz.

While this might seem like a limitation, it provides "accidental robustness" to narrowband data, such as legacy telephony recordings. Models trained on EnCodec tokens demonstrated superior performance on narrowband test sets (e.g., NIST SRE-18) compared to Mel-spectrogram models, even when the training data was wideband.

Results and Performance Trade-offs

The experimental results indicate that discrete tokens are highly competitive, generally performing within 1% of the Mel-spectrogram gold standard.

Automatic Speech Recognition (ASR)

On the LibriSpeech "clean" dataset, EnCodec tokens achieved parity with Mel-spectrograms, both yielding a Word Error Rate (WER) of 2.12%. The choice of codec matters; EnCodec consistently outperformed the Descript Audio Codec (DAC) in ASR tasks.

Speaker Verification

In speaker verification tasks, the gap remains small. On the VoxCeleb1-O clean benchmark, Mel-spectrograms achieved a 2.2% Equal Error Rate (EER), while EnCodec-32 followed closely at 2.9%.

Comparison of Mel-spectrograms and EnCodec-32 tokens on Speaker Verification tasks, highlighting robustness on NIST-SRE 18.

Compression Efficiency

One of the most significant advantages of discrete tokens is data efficiency. The researchers found that audio can be compressed to just 3 kbps (using 4 codebooks) while maintaining respectable performance—a 20x reduction compared to the ~128 kbps typically required for Mel-spectrograms.

Conclusion and Future Directions

Discrete audio tokens represent a powerful alternative to traditional spectrograms, offering massive compression and inherent robustness to narrowband signals. While Mel-spectrograms still maintain a slight edge in noisy or multilingual environments, the gap is narrowing. Future research aims to develop "task-aware" tokenizers that optimize for feature preservation rather than just signal reconstruction.

Summary updated

BibTeX
@inproceedings{puvvada2024discrete,
  author = {Krishna C. Puvvada and Nithin Rao Koluguri and Kunal Dhawan and Jagadeesh Balam and Boris Ginsburg},
  title = {{Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=85PEpkNGfu},
}
Figure