Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis

Théodor Lemerle, Harrison Vanderbyl, Vaibhav Srivastav, Nicolas Obin, Axel RoebelSubmission to ICLR 2025 · Withdrawn

TL;DR

Lina-Speech is a compact (169M parameter) text-to-speech architecture that replaces traditional quadratic self-attention with Gated Linear Attention (GLA). By optimizing the recurrent layers' initial hidden states rather than prepending audio prompts, it achieves voice cloning and style adaptation in under 15 seconds—matching or outperforming baseline models up to 4× its size while freeing the entire context window for generation.

What Problem Does This Paper Address?

Current autoregressive neural codec language models (such as VALL-E or VoiceCraft) treat voice cloning as an in-context prompt continuation task: reference audio is prepended to the input sequence, and the model continues generating from there.

While effective, this paradigm suffers from critical limitations:

  1. Context Window Contention: Concatenating audio prompts eats up sequence length, forcing an awkward trade-off between giving the model enough speaker reference audio and reserving space to synthesize long sentences.
  2. Computational Inefficiency: Audio codecs operate at high frame rates. Standard self-attention scales quadratically ($O(N^2)$) with sequence length, making long audio context training expensive.
  3. Limited Multi-Sample Aggregation: Prompt continuation cannot easily integrate multiple disconnected speech snippets, different recording styles, or multi-session recordings into a single coherent speaker identity.

Methods

Lina-Speech tackles these challenges through two main architectural innovations:

  • GLA-Based Autoregressive Backbone: Lina-Speech replaces Transformer self-attention blocks in its audio encoder and decoder with Gated Linear Attention (GLA) layers. GLA maintains constant-size recurrent state matrices ($S_t$) with data-dependent gating/decay: $$S_t = G_t \odot S_{t-1} + k_t^\top v_t$$ This allows $O(N)$ linear-time pretraining and constant-memory $O(1)$ inference per step, embedding an inductive bias toward locality that naturally fits speech generation.

  • Encoder-Decoder Conditioning: Text is encoded using a 6-block non-causal Transformer with RoPE. The audio decoder attends to text representations via PACA cross-attention (incorporating convolutional positional embeddings). The model predicts discrete acoustic tokens generated by WavTokenizer (4096 codebook size at 75 tokens/sec).

  • Initial-State Tuning (PEFT): Instead of prompt-continuation, Lina-Speech tunes the recurrent hidden state at step zero ($S_0$), which is ordinarily initialized to zero. For 5–30 minutes of target speaker data, this initial state can even be parameterized as a rank-1 matrix per head ($S_0 = k_0^\top v_0$). Tuning completes in around 20 optimization steps (<15 seconds on a consumer GPU) without modifying any model weights.
       Text Input ──► [ Text Encoder ] ──┐
                                         │ (Cross-Attention)
Audio: Init State S₀ ──► [ GLA Audio Encoder / Decoder ] ──► Codec Tokens
       (Optimized per speaker)

Datasets

  • Pre-training:
  • MLS (Multilingual LibriSpeech) English: ~10,000 hours of LibriVox audio, re-transcribed using an ASR model (NVIDIA NeMo FastConformer).
  • LibriTTS & LibriTTS-R: Clean multi-speaker datasets with normalized transcripts.
  • LibriLight-medium: Used for extended training on long-sequence generation.
  • Evaluation & Adaptation:
  • LibriTTS (Clean test split): In-domain zero-shot evaluation.
  • EXPRESSO: Expressive speech dataset used as an out-of-domain benchmark to test adaptation across diverse styles and emotional inflections.

Key Insights and Ideas

  • Linear Attention Fits Speech Better Than Transformers: Speech generation strongly relies on local temporal context and monotonic alignment rather than distant semantic recall. GLA's decay mechanism serves as a natural inductive bias, avoiding the waste of dense full-attention matrices.
  • Recurrent States as Expressive Speaker Embeddings: In linear RNN/GLA models, the hidden state acts as an accumulated memory of the prompt. Directly learning $S_0$ through gradient descent behaves like prompt tuning, but operates purely in state-space—leaving the sequence length and generation window entirely unaffected.
  • Rank-1 State Parameterization: The initial state matrix doesn't require full rank; factorizing it into outer products ($k_0^\top v_0$) retains high fidelity while minimizing trainable parameters and preventing overfitting on small target datasets.

Key Results

  • Zero-Shot Voice Cloning (LibriTTS):
  • Lina-Speech (169M parameters) achieved 4.18 Naturalness MOS (N-MOS) and 4.10 Similarity MOS (S-MOS), significantly outperforming larger LM-based baselines like VoiceCraft (833M: 3.62 N-MOS / 3.55 S-MOS) and VALL-E-X (300M: 3.21 N-MOS / 3.07 S-MOS).
  • It maintained competitive objective metrics: 7.5% Word Error Rate (WER) and 0.93 speaker cosine similarity (via WavLM).
  • Initial State Tuning vs. Prompt-Based Control:
  • In-Domain (LibriTTS): Lina-Speech outperformed Parler-TTS Mini (880M), reaching a lower WER (2.9% vs. 4.4%), higher speaker similarity (0.94 vs. 0.90), and higher subjective similarity (3.97 vs. 3.45 S-MOS).
  • Out-of-Domain (EXPRESSO): Adapted robustly to expressive speech styles with 3.6% WER and 0.93 similarity (compared to Parler-TTS's 4.9% WER and 0.87 similarity).
  • Adaptation required less than 15 seconds on a single NVIDIA RTX 3080.

Limitations and Future Work

  • Language Coverage: Pretraining and testing were restricted entirely to English audio corpora.
  • Model Scale: The authors evaluated only at 169M parameters; exploring whether GLA scales effectively to multi-billion parameter speech models remains an open question.
  • Non-Speech Vocalizations: The evaluation omitted non-verbal audio features (e.g., laughter, breathing tokens) present in expressive datasets like EXPRESSO.
  • Low Codec Frame-Rate Dependency: Performance heavily leverages WavTokenizer's aggressive compression (75 Hz); sensitivity to multi-codebook architectures (e.g., standard EnCodec RVQ) at scale requires further analysis.

Summary updated

Abstract

Neural codec language models have demonstrated state-of-the-art performance in text-to-speech (TTS) synthesis. Leveraging scalable architectures like autoregressive transformers, they capitalize on the availability of large speech datasets. When framing voice cloning as a prompt continuation task, these models excel at cloning voices from short audio samples. However this approach can't be extended to multiple speech excerpts and is limited since the concatenation of source and target speech must fall within the maximum context length which is determined during training. In this work, we propose a model that replaces transformers with emergent recurrent architecture such as Gated Linear Attention (GLA). Our model, Lina-Speech, outperforms or matches the baseline models that are up to 4x it's size. We showcase intial-state tuning as a parameter-efficient fine-tuning technique that optimizes the initial state of the recurrent layers, resulting in compact and expressive speaker embedding with fine-grained control over the speech style. Compared to prompt continuation, it allows voice cloning from multiple speech excerpts and full usage of the context window for synthesis. This approach is fast, deployable and does not rely on auxiliary modules. It also demonstrates extensive adaptation to out-of-domain data. We will release publicly our code and checkpoints. Audio samples are available at \url{https://anonymsubm.github.io}.
BibTeX
@misc{lemerle2024lina,
  author = {Théodor Lemerle and Harrison Vanderbyl and Vaibhav Srivastav and Nicolas Obin and Axel Roebel},
  title = {{Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis}},
  year = {2024},
  eprint = {2410.23320},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.23320},
}
Figure