Self-Attention Linguistic-Acoustic Decoder

Santiago Pascual, Antonio Bonafonte, Joan Serrà

Introduction

Published in late 2018, this research represents a pivotal shift in Text-to-Speech (TTS) by applying the Transformer architecture to acoustic modeling. The "acoustic model" is the critical pipeline stage that converts linguistic features into spectral data for sound generation. Traditionally dominated by Recurrent Neural Networks (RNNs), this stage faced significant latency challenges. The authors introduce SALAD (Self-Attention Linguistic-Acoustic Decoder), a Transformer-based model designed to overcome the sequential bottlenecks of RNNs, achieving a 10x speedup in inference time on CPUs.

The Sequential Bottleneck in TTS

Standard TTS systems predict acoustic features (pitch, spectral envelopes) for every frame of speech. For years, Long Short-Term Memory (LSTM) networks were the standard due to their ability to model sequences. However, RNNs are inherently sequential: calculating the $N$-th frame requires the previous $N-1$ frames. This dependency makes them slow to train and difficult to deploy on resource-constrained hardware like mobile devices or smart home assistants.

Methods: The SALAD Architecture

The core innovation of SALAD is the replacement of recurrent layers with Self-Attention modules. Unlike traditional decoders, the model structure is closer to a Transformer Encoder, processing the entire input sequence in parallel.

Comparison between the baseline RNN architecture and the proposed SALAD architecture.

Key components include:

  • Multi-Head Attention (MHA): Enables the model to simultaneously attend to different parts of the linguistic input (e.g., specific phonemes or stress markers) relevant to a given acoustic frame.
  • Positional Encodings: Sinusoidal time stamps provide the model with a sense of sequence order, which is not inherent in attention mechanisms.
  • Linguistic-Acoustic Mapping: The model maps 362-dimensional linguistic/prosodic features to 43 acoustic features, including Mel-frequency cepstral coefficients (MFCCs) and fundamental frequency (F0).

Experimental Setup

The researchers utilized the TCSTAR project dataset, focusing on one male (M1) and one female (F1) speaker. The dataset consists of approximately 100 minutes of training data per speaker, derived from parliamentary speeches and broadcast news.

Key Results and Performance

The primary advantage of SALAD is its efficiency. By leveraging parallelism, the model significantly reduces CPU inference latency—a critical metric for real-world deployment.

Inference time comparison between RNN and SALAD models as a function of audio duration.
  • Inference Speed: SALAD is over an order of magnitude faster than the RNN baseline. For a 35-second audio clip, the RNN required over 60 seconds (slower than real-time), whereas SALAD completed the task in approximately 5.5 seconds.
  • Acoustic Quality: There is a minor "accuracy tax." Mel Cepstral Distortion (MCD) increased by only 0.1 to 0.3 dB compared to RNNs.
  • Pitch Dynamics: The model exhibited higher F0 (pitch) error. Self-attention tends to focus on the mean of the data, leading to a "flatter" or more monotone pitch distribution compared to the natural variance captured by RNNs.

Conclusion and Future Directions

SALAD demonstrates that self-attention can effectively replace recurrence in acoustic modeling, offering massive gains in inference speed with minimal loss in spectral quality. The main limitation remains the "peaky" pitch distribution, which can result in less expressive speech. Future work may involve integrating short-memory non-recurrent modules, such as 1D convolutions, to better capture local frame-to-frame dynamics while maintaining the global efficiency of the Transformer architecture.

Summary updated

Abstract

The conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models like recurrent neural networks. Despite the good performance of such models (in terms of low distortion in the generated speech), their recursive structure tends to make them slow to train and to sample from. In this work, we try to overcome the limitations of recursive structure by using a module based on the transformer decoder network, designed without recurrent connections but emulating them with attention and positioning codes. Our results show that the proposed decoder network is competitive in terms of distortion when compared to a recurrent baseline, whilst being significantly faster in terms of CPU inference time. On average, it increases Mel cepstral distortion between 0.1 and 0.3 dB, but it is over an order of magnitude faster on average. Fast inference is important for the deployment of speech synthesis systems on devices with restricted resources, like mobile phones or embedded systems, where speaking virtual assistants are gaining importance.
BibTeX
@inproceedings{pascual2018self,
  author = {Santiago Pascual and Antonio Bonafonte and Joan Serrà},
  title = {{Self-Attention Linguistic-Acoustic Decoder}},
  booktitle = {IberSPEECH},
  year = {2018},
  url = {https://openreview.net/forum?id=GERRVR6lBS},
}
Figure