Introduction
Published in late 2018, this research represents a pivotal shift in Text-to-Speech (TTS) by applying the Transformer architecture to acoustic modeling. The "acoustic model" is the critical pipeline stage that converts linguistic features into spectral data for sound generation. Traditionally dominated by Recurrent Neural Networks (RNNs), this stage faced significant latency challenges. The authors introduce SALAD (Self-Attention Linguistic-Acoustic Decoder), a Transformer-based model designed to overcome the sequential bottlenecks of RNNs, achieving a 10x speedup in inference time on CPUs.
The Sequential Bottleneck in TTS
Standard TTS systems predict acoustic features (pitch, spectral envelopes) for every frame of speech. For years, Long Short-Term Memory (LSTM) networks were the standard due to their ability to model sequences. However, RNNs are inherently sequential: calculating the $N$-th frame requires the previous $N-1$ frames. This dependency makes them slow to train and difficult to deploy on resource-constrained hardware like mobile devices or smart home assistants.
Methods: The SALAD Architecture
The core innovation of SALAD is the replacement of recurrent layers with Self-Attention modules. Unlike traditional decoders, the model structure is closer to a Transformer Encoder, processing the entire input sequence in parallel.
Key components include:
- Multi-Head Attention (MHA): Enables the model to simultaneously attend to different parts of the linguistic input (e.g., specific phonemes or stress markers) relevant to a given acoustic frame.
- Positional Encodings: Sinusoidal time stamps provide the model with a sense of sequence order, which is not inherent in attention mechanisms.
- Linguistic-Acoustic Mapping: The model maps 362-dimensional linguistic/prosodic features to 43 acoustic features, including Mel-frequency cepstral coefficients (MFCCs) and fundamental frequency (F0).
Experimental Setup
The researchers utilized the TCSTAR project dataset, focusing on one male (M1) and one female (F1) speaker. The dataset consists of approximately 100 minutes of training data per speaker, derived from parliamentary speeches and broadcast news.
Key Results and Performance
The primary advantage of SALAD is its efficiency. By leveraging parallelism, the model significantly reduces CPU inference latency—a critical metric for real-world deployment.
- Inference Speed: SALAD is over an order of magnitude faster than the RNN baseline. For a 35-second audio clip, the RNN required over 60 seconds (slower than real-time), whereas SALAD completed the task in approximately 5.5 seconds.
- Acoustic Quality: There is a minor "accuracy tax." Mel Cepstral Distortion (MCD) increased by only 0.1 to 0.3 dB compared to RNNs.
- Pitch Dynamics: The model exhibited higher F0 (pitch) error. Self-attention tends to focus on the mean of the data, leading to a "flatter" or more monotone pitch distribution compared to the natural variance captured by RNNs.
Conclusion and Future Directions
SALAD demonstrates that self-attention can effectively replace recurrence in acoustic modeling, offering massive gains in inference speed with minimal loss in spectral quality. The main limitation remains the "peaky" pitch distribution, which can result in less expressive speech. Future work may involve integrating short-memory non-recurrent modules, such as 1D convolutions, to better capture local frame-to-frame dynamics while maintaining the global efficiency of the Transformer architecture.
Summary updated