Neural Text-to-Speech (TTS) has come a long way, moving from robotic drones to voices that are nearly indistinguishable from humans. However, if you’ve ever used a modern TTS system and heard it suddenly skip a word, repeat a syllable, or cut off the end of a sentence, you’ve encountered the "robustness" problem.
In their paper "FeatherTTS: Robust and Efficient Attention Based Neural TTS," researchers from Tencent, the Harbin Institute of Technology, and Intel introduce a system designed to eliminate these glitches while significantly boosting synthesis speed.
TL;DR
FeatherTTS is a high-stability neural TTS system that replaces traditional, error-prone attention mechanisms with a novel Gaussian attention and an attentive stop loss. By combining these with block sparsity in the decoder, the authors achieved a 3.5x speedup over Tacotron2 and a massive reduction in word error rates, making it 35x faster than real-time on a single CPU.
What Problem Does This Paper Address?
Most modern TTS systems (like Tacotron2) use an attention mechanism to align the input text with the output audio frames. While elegant, these mechanisms have two major flaws:
- Instability: Content-based attention often "loses its place," leading to word skipping or repeating, especially on "hard" texts or long sentences. Even GMM-based (Gaussian Mixture Model) attention can suffer from alignment "spikes" because it isn't strictly monotonic or normalized.
- Efficiency: These models are typically autoregressive (generating one frame at a time), which is computationally expensive and slow to deploy in production environments.
- The "Early Stop" Bug: Standard models use a "stop token" to decide when to finish. For complex sentences, this token often triggers too early, cutting off the speaker mid-sentence.
Methods
The researchers proposed three key innovations to transform the standard Tacotron2 architecture into FeatherTTS:
1. Gaussian Attention
Instead of using a complex mixture of distributions, FeatherTTS uses a single, simple Gaussian attention mechanism. Crucially, they enforce strict monotonicity. The "center" of the attention (the mean, $\mu$) is calculated by adding a non-negative offset to the previous step's mean. This ensures the model must move forward through the text and can never jump backward or skip ahead erratically.
2. Attentive Stop Loss (ATL)
To fix the early-stop issue, the authors ditched the traditional stop token entirely. Instead, they introduced the Attentive Stop Loss. This loss function forces the attention mean ($\mu$) to reach the very end of the phoneme sequence. During inference, the model simply stops when the attention mean reaches the end of the input. This ties the "end of speech" directly to the "end of text," making it much more reliable.
3. Sparse Autoregressive Decoder
To tackle speed, the team applied block sparsity to the LSTM layers in the decoder. They used a weight pruning scheme to reach 90% sparsity. By removing the least important weights, they reduced the computational load significantly without sacrificing the naturalness of the voice.
Datasets
The models were trained and evaluated using:
- Mandarin Corpus: 20 hours of high-quality recordings from a professional female broadcaster.
- Split: 18 hours for training, 2 hours for testing.
- Audio Specs: 24kHz sampling rate, 16-bit format, converted to 80-band mel-spectrograms.
- Vocoder: The mel-spectrograms were turned into waveforms using FeatherWave, a high-fidelity neural vocoder.
Key Insights and Ideas
- Constraints Breed Stability: By forcing the attention mechanism to be strictly monotonic and normalized, the researchers eliminated the "spiky" alignment issues that plague GMM-based models.
- Alignment is the Best Stop Signal: The most logical way to know when to stop speaking is to know you've reached the last letter of the text. Tying the stopping condition to the attention mean is more intuitive and robust than training a separate classifier (the stop token).
- Sparsity is a Free Lunch (Almost): The paper reinforces the idea that a large, sparse network often performs better than a small, dense one. Pruning 90% of the decoder weights actually resulted in a slight improvement in Mean Opinion Score (MOS), likely due to better generalization.
Key Results
The experimental results show that FeatherTTS is both "tougher" and faster than its predecessors:
- Robustness: In a test of 20 "hard" sentences, the Word Error Rate (WER) dropped from 4.1% (Tacotron2 with GMMv2b) to just 0.9% with FeatherTTS.
- Naturalness: FeatherTTS achieved a Mean Opinion Score (MOS) of 4.33, slightly outperforming the baseline Tacotron2 (4.31).
- Speed:
- Acoustic feature generation is 3.5x faster than Tacotron2.
- On a single CPU core, it runs at 35x real-time.
- When optimized with Intel’s BF16 format, it reaches a staggering 60x real-time speed.
Limitations and Future Work
While FeatherTTS shows remarkable results, there are a few areas for further exploration:
- Language Generalization: The study focused exclusively on Mandarin. While the principles of Gaussian attention are language-agnostic, testing on languages with different rhythmic structures (like English or stress-timed languages) would be a valuable next step.
- Edge Deployment: The authors mention that the speedups make edge deployment possible, but the paper does not provide specific benchmarks for mobile or IoT processors.
- Future Direction: The team plans to specifically investigate the performance of FeatherTTS on low-power edge devices.
Summary updated