Low Rank Factorization for Compact Multi-Head Self-Attention

Sneha Mehta, Huzefa Rangwala, Naren RamakrishnanarXiv 2019 · arXiv

Introduction

Transformer-based models like BERT and RoBERTa have set new benchmarks in Natural Language Processing (NLP). However, their reliance on the self-attention mechanism introduces a significant "computational tax." Standard self-attention has a quadratic complexity of $O(n^2 \cdot d)$ relative to sequence length $n$, making it computationally expensive for long documents and difficult to deploy on resource-constrained devices.

This paper introduces LAMA (Low-rank factorizAtion based Multi-head Attention), a lightweight alternative designed for efficiency. LAMA achieves state-of-the-art performance on text classification while maintaining a linear scaling factor and high parameter efficiency.

Methodology: The LAMA Architecture

LAMA streamlines the attention process by moving away from dense word-to-word querying. Instead, it employs a "global-to-local" approach to identify salient features within a document.

The LAMA architecture, featuring a Bi-GRU backbone and a low-rank multi-head attention mechanism that queries a global context vector.

The core technical innovations include:

  1. Global Context Querying: Rather than every word attending to every other word, LAMA computes attention scores by querying a single global context vector (initialized as the mean of word embeddings). This reduces complexity from $O(n^2)$ to $O(n)$.
  2. Low-Rank Bilinear Pooling: To support multiple attention heads without a massive parameter explosion, the authors use a low-rank matrix factorization trick. The bilinear projection matrix is factorized into two low-rank matrices ($P$ and $Q$). A Hadamard product (element-wise multiplication) then produces multiple attention distributions efficiently.
  3. Bi-GRU Backbone: A Bi-directional Gated Recurrent Unit (bi-GRU) captures sequential dependencies before the attention layer is applied.
  4. Disagreement Regularization: An auxiliary loss term penalizes heads for being too similar, ensuring that different attention heads capture distinct "aspects" of the text.

Experimental Setup

The model was evaluated across several diverse text classification benchmarks, including:

  • Yelp: Business ratings and sentiment analysis.
  • IMDB: Binary sentiment analysis for movie reviews.
  • News Aggregator & Reuters (r8): Topic classification for news articles.

Key Results and Efficiency

LAMA demonstrated performance comparable to or exceeding strong baselines like CNNs, Bi-GRUs, and standard Self-Attention Networks (SAN). Notably, LAMA outperformed BERT-base on several datasets while being significantly faster to train and having a smaller memory footprint.

Comparison of inference time vs. input length on IMDB and Reuters datasets. LAMA (blue) scales linearly and remains significantly faster than the Transformer Encoder (green).

Parameter and Computational Efficiency

LAMA is approximately three times more parameter-efficient than a standard Transformer Encoder. Because of the low-rank trick, increasing the number of attention heads only marginally impacts the total parameter count. In contrast, standard Transformers scale linearly with the number of heads. For classification tasks, performance generally peaked around 15 attention heads.

Interpretability

The global context query makes attention weights highly interpretable. The model effectively identifies key sentiment-bearing words (e.g., "amazing") and correctly adjusts their importance based on the surrounding context (e.g., "not amazing").

Conclusion and Future Work

LAMA proves that for many supervised tasks like classification, the full complexity of word-to-word attention is often redundant. By utilizing global-to-local querying and low-rank factorization, LAMA provides a path toward high-performance NLP on edge devices.

Future research aims to address the sequential bottleneck of the Bi-GRU backbone by exploring purely non-recurrent architectures with positional embeddings to regain the parallelization benefits of Transformers. Additionally, the authors suggest "phrase-level" queries as a potential middle ground to further enhance document representation.

Summary updated

Abstract

Effective representation learning from text has been an active area of research in the fields of NLP and text mining. Attention mechanisms have been at the forefront in order to learn contextual sentence representations. Current state-of-the-art approaches for many NLP tasks use large pre-trained language models such as BERT, XLNet and so on for learning representations. These models are based on the Transformer architecture that involves recurrent blocks of computation consisting of multi-head self-attention and feedforward networks. One of the major bottlenecks largely contributing to the computational complexity of the Transformer models is the self-attention layer, that is both computationally expensive and parameter intensive. In this work, we introduce a novel multi-head self-attention mechanism operating on GRUs that is shown to be computationally cheaper and more parameter efficient than self-attention mechanism proposed in Transformers for text classification tasks. The efficiency of our approach mainly stems from two optimizations; 1) we use low-rank matrix factorization of the affinity matrix to efficiently get multiple attention distributions instead of having separate parameters for each head 2) attention scores are obtained by querying a global context vector instead of densely querying all the words in the sentence. We evaluate the performance of the proposed model on tasks such as sentiment analysis from movie reviews, predicting business ratings from reviews and classifying news articles into topics. We find that the proposed approach matches or outperforms a series of strong baselines and is more parameter efficient than comparable multi-head approaches. We also perform qualitative analyses to verify that the proposed approach is interpretable and captures context-dependent word importance.
BibTeX
@misc{mehta2019low,
  author = {Sneha Mehta and Huzefa Rangwala and Naren Ramakrishnan},
  title = {{Low Rank Factorization for Compact Multi-Head Self-Attention}},
  year = {2019},
  eprint = {1912.00835},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.00835},
}
Figure