Factorized Self-Supervised Speech Tokenization

Benjamin Van Niekerk, Jean-Philippe Letendre, Nicol Visser, Hugo Seute, Herman Kamper, Mirco RavanelliarXiv 2026 · arXiv

TL;DR

Facto is a new speech tokenizer that uses linear factorization to disentangle "what is said" (content and prosody) from "who said it" (speaker identity and recording conditions). By stripping away speaker-specific noise before discretization, Facto produces highly stable tokens that significantly improve performance in text-to-speech, voice conversion, and spoken language modeling.

What Problem Does This Paper Address?

In the world of Generative AI for audio, speech tokenizers are the bridge between continuous sound waves and the discrete "vocabulary" that Large Language Models (LLMs) understand. However, most modern tokenizers are "messy." When they convert audio into discrete units, they often pack everything—phonemes, emotional delivery, speaker timbre, and background noise—into a single stream.

This creates a massive redundancy problem. If a tokenizer represents the word "hello" differently for every possible speaker, the downstream LLM has to learn every combination of speaker and content. This requires enormous codebooks and makes models brittle; a tiny bit of background noise can flip a token, causing the model to hallucinate or lose coherence. The authors of Facto ask: Can we build a tokenizer that focuses strictly on the time-varying content, making it more robust and efficient?

Methods

Facto builds on the idea of linear factorization of self-supervised learning (SSL) features (specifically from WavLM). The process involves three main stages:

  1. Factorization: The researchers take a fixed set of training speakers and align their speech frames using nearest neighbors. They then solve an optimization problem to decompose these features into a shared content matrix ($C$) and speaker-specific transformations ($S_k$). Essentially, they find the "common denominator" of what is being said across different voices.
  2. Content Estimation & Clustering: To handle speakers the model has never heard before, they fit a simple linear least-squares estimator. This projects raw SSL features from unseen speech directly into the "content space." These projected features are then patched (stacked), decorrelated via PCA, and clustered using $k$-means to create the final discrete tokens.
  3. Decoding: To turn tokens back into audio, they use a lightweight decoder based on HiFiGAN. Crucially, they re-inject speaker identity at this stage using FiLM (Feature-wise Linear Modulation) layers, which modulate the decoder's activations based on a separate speaker embedding.

Datasets

The authors utilized a robust suite of standard speech datasets:

  • LibriSpeech: Used for the core factorization, resynthesis evaluation, and ASR probing.
  • LibriLight (6k and 60k hours): Used for training the spoken language models.
  • LibriHeavy: A large-scale dataset used specifically for training the text-to-speech (TTS) models.
  • Expresso: Used to test how well the tokenizer handles expressive and spontaneous speech compared to "read" speech.
  • Seed-TTS-eval: A benchmark for evaluating the quality and similarity of generated speech.
  • VoxCeleb1: Used for "probing" experiments to verify that speaker information was successfully removed from the content tokens.

Key Insights and Ideas

  • Linearity is Sufficient: One of the most surprising takeaways is that a linear estimator is highly effective at disentangling speech. While many researchers reach for complex adversarial neural networks to achieve disentanglement, Facto shows that the underlying structure of SSL features like WavLM allows for a much simpler, more interpretable mathematical approach.
  • The Power of Stability: The authors introduce Unit Edit Distance (UED) to measure token stability. They found that by removing static speaker components before clustering, Facto’s tokens are roughly 50% more stable under noise than competing methods.
  • Theoretical Scaling Laws: The paper provides a rare theoretical analysis of factorization. They prove that the factorization error decays at a rate of $O(1/\sqrt{K})$ relative to the number of training speakers ($K$), while the content estimation error scales at $O(1/K)$. This gives a mathematical "green light" for scaling: more speaker diversity directly leads to more robust representations.

Key Results

Facto was benchmarked against state-of-the-art tokenizers like FocalCodec and Kanade:

  • Noise Robustness: Facto achieved significantly lower UED across various noise profiles (White, Pink, Brown, etc.). For example, at 12.5Hz, Facto had an average UED of ~20-22, compared to ~48-50 for baselines.
  • Text-to-Speech: In TTS tasks on the Seed-TTS set, Facto (12.5Hz) achieved a Word Error Rate (WER) of 2.3%, vastly outperforming Kanade (4.5%) and FocalCodec (3.6%). It also scored higher in subjective naturalness and speaker similarity.
  • Voice Conversion: Facto showed high speaker similarity (SIM scores of 0.94-0.95), proving that the decoder successfully re-integrates speaker identity even when the tokens themselves are speaker-agnostic.

Limitations and Future Work

  • Feature Dependency: The current implementation relies on layer six of WavLM. The authors note that it remains an open question how well this generalizes to other SSL models or different layers.
  • Global Speaker Embeddings: Facto uses a single global embedding for the entire utterance. This means it might struggle to capture subtle shifts in a speaker's voice or recording conditions that happen during a long sentence.
  • Bitrate Overhead: While the content tokens are compact, the system requires transmitting a speaker embedding for reconstruction. For very short utterances, this embedding can significantly increase the total bitrate.
  • Future Direction: The authors suggest exploring "local" speaker embeddings extracted over shorter windows to better capture dynamic speaker characteristics.

Summary updated

Abstract

Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto's convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling. Code and checkpoints at https://github.com/ubisoft/ubisoft-laforge-facto/ and demo page at https://ubisoft-laforge.github.io/speech/facto/.
BibTeX
@misc{niekerk2026factorized,
  author = {Benjamin Van Niekerk and Jean-Philippe Letendre and Nicol Visser and Hugo Seute and Herman Kamper and Mirco Ravanelli},
  title = {{Factorized Self-Supervised Speech Tokenization}},
  year = {2026},
  eprint = {2611.18898},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18898},
}

Loading related content...

Figure