Polynomial Mixing for Efficient Self-supervised Speech Encoders

Eva Feillet, Ryan Whetten, David Picard, Alexandre Allauzen

TL;DR

Researchers have introduced the Polynomial Mixer (PoM), a new "drop-in" replacement for self-attention in speech recognition models. By using a polynomial-based global state instead of pairwise token interactions, PoM achieves linear complexity ($O(n)$), making it significantly faster and more memory-efficient than standard Transformers for long audio sequences while maintaining competitive accuracy.

What Problem Does This Paper Address?

State-of-the-art speech-to-text models (like Whisper or wav2vec 2.0) rely on the Transformer architecture. At the heart of the Transformer is Multi-Head Self-Attention (MHA), which compares every token in a sequence to every other token.

While powerful, MHA has a "quadratic bottleneck": if you double the length of your audio input, the computational cost and memory usage quadruple. This is a massive problem for speech processing because audio signals are naturally long (a few seconds of audio translates into hundreds or thousands of "tokens"). While researchers have found ways to make attention more efficient in NLP, speech recognition has unique challenges—specifically, the need to process very long sequences where local acoustic information is often more important than distant global dependencies.

Methods

The authors propose the Polynomial Mixer (PoM). Instead of calculating a massive $N \times N$ attention matrix, PoM summarizes the entire sequence into a single global state representation $H(X)$.

Here’s how it works:

  1. Polynomial Branches: The input is projected through $k$ different "branches." These are combined using element-wise (Hadamard) products to create a polynomial representation of degree $k$.
  2. Global Summation: All processed tokens are summed into a single global vector. This "summary" captures the essence of the whole sequence.
  3. Token-wise Selection: Each individual token then uses a "selector" (a learned query) to pull the relevant information it needs from that global summary.
  4. Linear Complexity: Because every token interacts with a single global state rather than every other token, the complexity stays linear relative to the input length.

The authors integrated PoM into a Conformer architecture (a popular speech model that mixes convolutions and attention) as a direct replacement for the MHA layers.

Datasets

The models were evaluated using the industry-standard LibriSpeech corpus:

  • Pre-training: LibriSpeech-960h (960 hours of unlabeled English audiobooks) using the BEST-RQ self-supervised learning scheme.
  • Fine-tuning/Evaluation: LibriSpeech-100h "clean" subset for downstream Automatic Speech Recognition (ASR) tasks.
  • Benchmarks: Performance was measured on the test-clean and test-other subsets.

Key Insights and Ideas

  • Expressivity over Averaging: PoM is essentially a more sophisticated version of "SummaryMixing." While SummaryMixing just takes a simple average of tokens, PoM uses higher-order polynomial interactions, allowing it to capture more complex relationships in the audio data.
  • The "Local" Advantage: In speech, you often don't need to know what happened at minute 1 to understand a phoneme at minute 5. PoM exploits this by providing a global context that tokens can dip into, without the overhead of exhaustive pairwise comparisons.
  • Drop-in Compatibility: One of the biggest strengths of PoM is that it doesn't require a total architectural overhaul. It fits right into existing Transformer/Conformer blocks.
  • Layer Drop is Essential: The researchers found that "Layer Drop" (randomly skipping layers during training) improved performance for all mixer types, including PoM.

Key Results

  • Efficiency: For an 80-second audio input, PoM used 2.8 times less memory than the standard "RelPosMHA" (Relative Positional MHA).
  • Speed: PoM demonstrated a runtime close to the simplest linear mixers and was faster than optimized RoPE (Rotary Positional Embedding) implementations.
  • Accuracy: On the LibriSpeech test-clean task, the 95M parameter PoM model achieved a Word Error Rate (WER) of 8.31% (without a language model), outperforming other linear alternatives like SummaryMixing (9.79%) and FastFormer (9.32%), and coming close to regular MHA (8.59%).
  • Scalability: As input length increases from 10 to 80 seconds, PoM’s memory and time usage grow linearly, while MHA’s usage explodes quadratically (see Figure 2 in the paper).

Limitations and Future Work

While PoM is highly efficient and beats other linear mixers, it still slightly trails the very best (but computationally expensive) attention variants like RelPosMHA in terms of pure Word Error Rate.

The authors suggest several paths forward:

  • Hybrid Architectures: Using standard MHA in the early layers (to capture complex local features) and PoM in the upper layers (where attention maps tend to be simpler/diagonal anyway).
  • Hardware Optimization: Further optimizing the implementation for embedded and edge devices where memory is at a premium.
  • Broader Tasks: Testing PoM on non-ASR tasks like emotion recognition or speaker verification to see if the polynomial summary holds up across different speech domains.

Summary updated

Abstract

State-of-the-art speech-to-text models typically employ Transformer-based encoders that model token dependencies via self-attention mechanisms. However, the quadratic complexity of self-attention in both memory and computation imposes significant constraints on scalability. In this work, we propose a novel token-mixing mechanism, the Polynomial Mixer (PoM), as a drop-in replacement for multi-head self-attention. PoM computes a polynomial representation of the input with linear complexity with respect to the input sequence length. We integrate PoM into a self-supervised speech representation learning framework based on BEST-RQ and evaluate its performance on downstream speech recognition tasks. Experimental results demonstrate that PoM achieves a competitive word error rate compared to full self-attention and other linear-complexity alternatives, offering an improved trade-off between performance and efficiency in time and memory.
BibTeX
@misc{feillet2026polynomial,
  author = {Eva Feillet and Ryan Whetten and David Picard and Alexandre Allauzen},
  title = {{Polynomial Mixing for Efficient Self-supervised Speech Encoders}},
  year = {2026},
  eprint = {2603.00683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.00683},
}
Figure