XNOR-FORMER: Learning Accurate Approximations in Long Speech Transformers

Roshan Sharma, Bhiksha RajarXiv 2022 · DOI · arXiv

Introduction

Transformers have become the backbone of modern AI, yet they face a significant scalability challenge in speech processing. The standard self-attention mechanism exhibits quadratic complexity ($O(N^2)$) relative to sequence length $N$. In speech, where a few seconds of audio can generate thousands of feature vectors, this bottleneck limits the model's ability to handle long-form content. While linear transformers exist, they often sacrifice accuracy for efficiency. The XNOR-former addresses this gap by introducing a novel linear attention mechanism that approximates the softmax function with high fidelity.

The XNOR Approximation Method

The core innovation of the XNOR-former is the XNOR Self-Attention mechanism. This approach is rooted in a mathematical observation regarding the sigmoid function, $\sigma(xy)$, which behaves similarly to a logical XNOR gate. The authors propose an approximation: $$\sigma(xy) \approx \sigma(x)\sigma(y) + (1-\sigma(x))(1-\sigma(y))$$

Visual comparison of the sigmoid product $\sigma(xy)$ and its XNOR-based decomposition.

This decomposition is critical because it transforms a non-linear operation on a product into a sum of products of individual terms. By factorizing the computation this way, the model can leverage the associative law of matrix multiplication to compute attention in linear time ($O(N)$) without ever materializing the massive $N \times N$ attention matrix.

To further enhance performance, the authors introduced Weighted-XNOR (W-XNOR), which adds learnable parameters to the decomposition, allowing the model to optimize the approximation during training. Additionally, the architecture integrates factorable positional encodings, such as Rotary Positional Encodings (RoPE), ensuring the model remains sensitive to the sequential nature of speech data.

Experimental Evaluation

The XNOR-former was benchmarked against several speech-centric datasets, including Librispeech-100, INTERVIEW-300 (NPR conversations), and How2 (instructional video summarization).

Accuracy and Performance

On Librispeech-100, the W-XNOR-former achieved a Word Error Rate (WER) of 8.1% on the "test-clean" set, significantly outperforming the previous state-of-the-art linear transformer, Cosformer (10.8% WER). In speech summarization tasks on the How2 dataset, it surpassed the Cosformer by 5 points on the ROUGE-L metric, demonstrating superior long-range context capture.

Computational Efficiency

The primary advantage of the XNOR-former is its scaling behavior. As shown in the performance analysis, standard softmax attention compute time grows quadratically, becoming prohibitive as sequences reach 30,000 frames. In contrast, the XNOR-former maintains efficient, linear compute time.

Attention compute time comparison showing the quadratic explosion of standard softmax vs. the linear scaling of XNOR-former and other linear variants.

Conclusion

The XNOR-former successfully bridges the gap between the efficiency of linear transformers and the accuracy of full-attention models. By utilizing a clever XNOR-like decomposition of the sigmoid function, it enables the processing of extremely long speech sequences in $O(N)$ time. This architecture proves particularly effective for complex speech tasks like long-form recognition and summarization, where capturing distal context is essential.

Summary updated

Abstract

Transformers are among the state of the art for many tasks in speech, vision, and natural language processing, among others. Self-attentions, which are crucial contributors to this performance have quadratic computational complexity, which makes training on longer input sequences challenging. Prior work has produced state-of-the-art transformer variants with linear attention, however, current models sacrifice performance to achieve efficient implementations. In this work, we develop a novel linear transformer by examining the properties of the key-query product within self-attentions. Our model outperforms state of the art approaches on speech recognition and speech summarization, resulting in 1 % absolute WER improvement on the Librispeech-100 speech recognition benchmark and a new INTERVIEW speech recognition benchmark, and 5 points on ROUGE for summarization with How2.
BibTeX
@misc{sharma2022xnor,
  author = {Roshan Sharma and Bhiksha Raj},
  title = {{XNOR-FORMER: Learning Accurate Approximations in Long Speech Transformers}},
  year = {2022},
  eprint = {2210.16643},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.16643},
}
Figure