TL;DR
Lina-Speech is a compact (169M parameter) text-to-speech architecture that replaces traditional quadratic self-attention with Gated Linear Attention (GLA). By optimizing the recurrent layers' initial hidden states rather than prepending audio prompts, it achieves voice cloning and style adaptation in under 15 seconds—matching or outperforming baseline models up to 4× its size while freeing the entire context window for generation.
What Problem Does This Paper Address?
Current autoregressive neural codec language models (such as VALL-E or VoiceCraft) treat voice cloning as an in-context prompt continuation task: reference audio is prepended to the input sequence, and the model continues generating from there.
While effective, this paradigm suffers from critical limitations:
- Context Window Contention: Concatenating audio prompts eats up sequence length, forcing an awkward trade-off between giving the model enough speaker reference audio and reserving space to synthesize long sentences.
- Computational Inefficiency: Audio codecs operate at high frame rates. Standard self-attention scales quadratically ($O(N^2)$) with sequence length, making long audio context training expensive.
- Limited Multi-Sample Aggregation: Prompt continuation cannot easily integrate multiple disconnected speech snippets, different recording styles, or multi-session recordings into a single coherent speaker identity.
Methods
Lina-Speech tackles these challenges through two main architectural innovations:
-
GLA-Based Autoregressive Backbone: Lina-Speech replaces Transformer self-attention blocks in its audio encoder and decoder with Gated Linear Attention (GLA) layers. GLA maintains constant-size recurrent state matrices ($S_t$) with data-dependent gating/decay: $$S_t = G_t \odot S_{t-1} + k_t^\top v_t$$ This allows $O(N)$ linear-time pretraining and constant-memory $O(1)$ inference per step, embedding an inductive bias toward locality that naturally fits speech generation.
-
Encoder-Decoder Conditioning: Text is encoded using a 6-block non-causal Transformer with RoPE. The audio decoder attends to text representations via PACA cross-attention (incorporating convolutional positional embeddings). The model predicts discrete acoustic tokens generated by WavTokenizer (4096 codebook size at 75 tokens/sec).
- Initial-State Tuning (PEFT): Instead of prompt-continuation, Lina-Speech tunes the recurrent hidden state at step zero ($S_0$), which is ordinarily initialized to zero. For 5–30 minutes of target speaker data, this initial state can even be parameterized as a rank-1 matrix per head ($S_0 = k_0^\top v_0$). Tuning completes in around 20 optimization steps (<15 seconds on a consumer GPU) without modifying any model weights.
Text Input ──► [ Text Encoder ] ──┐
│ (Cross-Attention)
Audio: Init State S₀ ──► [ GLA Audio Encoder / Decoder ] ──► Codec Tokens
(Optimized per speaker)
Datasets
- Pre-training:
- MLS (Multilingual LibriSpeech) English: ~10,000 hours of LibriVox audio, re-transcribed using an ASR model (NVIDIA NeMo FastConformer).
- LibriTTS & LibriTTS-R: Clean multi-speaker datasets with normalized transcripts.
- LibriLight-medium: Used for extended training on long-sequence generation.
- Evaluation & Adaptation:
- LibriTTS (Clean test split): In-domain zero-shot evaluation.
- EXPRESSO: Expressive speech dataset used as an out-of-domain benchmark to test adaptation across diverse styles and emotional inflections.
Key Insights and Ideas
- Linear Attention Fits Speech Better Than Transformers: Speech generation strongly relies on local temporal context and monotonic alignment rather than distant semantic recall. GLA's decay mechanism serves as a natural inductive bias, avoiding the waste of dense full-attention matrices.
- Recurrent States as Expressive Speaker Embeddings: In linear RNN/GLA models, the hidden state acts as an accumulated memory of the prompt. Directly learning $S_0$ through gradient descent behaves like prompt tuning, but operates purely in state-space—leaving the sequence length and generation window entirely unaffected.
- Rank-1 State Parameterization: The initial state matrix doesn't require full rank; factorizing it into outer products ($k_0^\top v_0$) retains high fidelity while minimizing trainable parameters and preventing overfitting on small target datasets.
Key Results
- Zero-Shot Voice Cloning (LibriTTS):
- Lina-Speech (169M parameters) achieved 4.18 Naturalness MOS (N-MOS) and 4.10 Similarity MOS (S-MOS), significantly outperforming larger LM-based baselines like VoiceCraft (833M: 3.62 N-MOS / 3.55 S-MOS) and VALL-E-X (300M: 3.21 N-MOS / 3.07 S-MOS).
- It maintained competitive objective metrics: 7.5% Word Error Rate (WER) and 0.93 speaker cosine similarity (via WavLM).
- Initial State Tuning vs. Prompt-Based Control:
- In-Domain (LibriTTS): Lina-Speech outperformed Parler-TTS Mini (880M), reaching a lower WER (2.9% vs. 4.4%), higher speaker similarity (0.94 vs. 0.90), and higher subjective similarity (3.97 vs. 3.45 S-MOS).
- Out-of-Domain (EXPRESSO): Adapted robustly to expressive speech styles with 3.6% WER and 0.93 similarity (compared to Parler-TTS's 4.9% WER and 0.87 similarity).
- Adaptation required less than 15 seconds on a single NVIDIA RTX 3080.
Limitations and Future Work
- Language Coverage: Pretraining and testing were restricted entirely to English audio corpora.
- Model Scale: The authors evaluated only at 169M parameters; exploring whether GLA scales effectively to multi-billion parameter speech models remains an open question.
- Non-Speech Vocalizations: The evaluation omitted non-verbal audio features (e.g., laughter, breathing tokens) present in expressive datasets like EXPRESSO.
- Low Codec Frame-Rate Dependency: Performance heavily leverages WavTokenizer's aggressive compression (75 Hz); sensitivity to multi-codebook architectures (e.g., standard EnCodec RVQ) at scale requires further analysis.
Summary updated