Polynomial Mixing for Efficient Self-supervised Speech Encoders

Eva Feillet, Ryan Whetten, David Picard, Alexandre Allauzen

Abstract

State-of-the-art speech-to-text models typically employ Transformer-based encoders that model token dependencies via self-attention mechanisms. However, the quadratic complexity of self-attention in both memory and computation imposes significant constraints on scalability. In this work, we propose a novel token-mixing mechanism, the Polynomial Mixer (PoM), as a drop-in replacement for multi-head self-attention. PoM computes a polynomial representation of the input with linear complexity with respect to the input sequence length. We integrate PoM into a self-supervised speech representation learning framework based on BEST-RQ and evaluate its performance on downstream speech recognition tasks. Experimental results demonstrate that PoM achieves a competitive word error rate compared to full self-attention and other linear-complexity alternatives, offering an improved trade-off between performance and efficiency in time and memory.
BibTeX
@misc{feillet2026polynomial,
  author = {Eva Feillet and Ryan Whetten and David Picard and Alexandre Allauzen},
  title = {{Polynomial Mixing for Efficient Self-supervised Speech Encoders}},
  year = {2026},
  eprint = {2603.00683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.00683},
}

Loading related content...

Figure