Abstract
Historically, most speech models in machine-learning have used the mel-spectrogram as a speech representation. Recently, discrete audio tokens produced by neural audio codecs have become a popular alternate speech representation for speech synthesis tasks such as text-to-speech (TTS). However, the data distribution produced by such codecs is too complex for some TTS models to predict, typically requiring large autoregressive models to get good quality. Most existing audio codecs use Residual Vector Quantization (RVQ) to compress and reconstruct the time-domain audio signal. We propose a spectral codec which uses Finite Scalar Quantization (FSQ) to compress the mel-spectrogram and reconstruct the time-domain audio signal. A study of objective audio quality metrics and subjective listening tests suggests that our spectral codec has comparable perceptual quality to equivalent audio codecs. We show that FSQ, and the use of spectral speech representations, can both improve the performance of parallel TTS models.
Introduction: The Quality vs. Speed Trade-off
In Text-to-Speech (TTS) research, a persistent challenge exists in balancing audio quality with generation speed. Traditional mel-spectrograms are easily predicted by fast, non-autoregressive (parallel) models like FastPitch, but they often result in "over-smoothed" or robotic audio. Conversely, modern neural audio codecs (e.g., EnCodec) produce high-fidelity sound by converting audio into discrete tokens. However, these codecs typically rely on Residual Vector Quantization (RVQ), which creates a complex hierarchical dependency between tokens. This hierarchy makes it extremely difficult for parallel models to predict all tokens simultaneously, often leading to "hallucinations" or unintelligible speech unless slow, step-by-step autoregressive (AR) models are used.
Proposed Method: Spectral Domain and FSQ
To bridge this gap, researchers have introduced Spectral Codecs. This approach shifts the encoding process from the raw time-domain waveform to the spectral domain (mel-spectrograms). This leverages the structured frequency data that parallel models already handle well.
The core innovation is the use of Finite Scalar Quantization (FSQ) instead of RVQ. Unlike the hierarchical nature of RVQ, FSQ maps values onto a fixed, multi-dimensional grid. This removes inter-dependencies between codebooks, allowing a model to predict all tokens for a single timestep in parallel without losing coherence.
The architecture utilizes a modified HiFi-GAN decoder and a residual encoder to manage the quantization process. For TTS tasks, the system integrates with FastPitch, a popular non-autoregressive transformer.
Experimental Setup
The codec was trained on a massive 12.8k-hour dataset of 44.1 kHz speech derived from Multilingual LibriSpeech (MLS) and LibriVox, filtered for full-bandwidth quality. TTS training utilized the HiFi-TTS dataset (300 hours, 10 speakers). Evaluation was conducted on a held-out MLS set featuring 42 speakers.
Key Results
The evaluation focused on reconstruction quality and TTS performance using both objective metrics and subjective Mean Opinion Scores (MOS).
Reconstruction and Intelligibility
Reconstruction tests showed that the Spectral FSQ codec (MOS 3.80) performs comparably to standard audio-based RVQ codecs (MOS 3.75), confirming that the spectral domain transition does not compromise base audio quality.
TTS Performance in Parallel Models
The primary advantage of FSQ is evident in parallel TTS settings. As shown in the results table, the Spectral FSQ codec enabled the FastPitch model to achieve high intelligibility and quality where traditional RVQ-based audio codecs failed.
Notably, the Spectral FSQ model achieved a Word Error Rate (WER) of 2.30%, nearly matching the ground truth (2.25%). In contrast, standard audio RVQ codecs performed so poorly in parallel configurations that they were excluded from MOS testing due to unintelligible output.
Discussion and Future Work
The findings suggest that FSQ acts as a regularizer, reducing the noise and over-smoothing typically found in traditional mel-spectrogram synthesis. While spectral codecs excel in parallel models, audio-based FSQ codecs still hold a slight edge in autoregressive models. Future research will focus on improving performance across diverse acoustic conditions by incorporating more robust context and reference audio.
Summary updated