Introduction
Transformer-based models like BERT and RoBERTa have set new benchmarks in Natural Language Processing (NLP). However, their reliance on the self-attention mechanism introduces a significant "computational tax." Standard self-attention has a quadratic complexity of $O(n^2 \cdot d)$ relative to sequence length $n$, making it computationally expensive for long documents and difficult to deploy on resource-constrained devices.
This paper introduces LAMA (Low-rank factorizAtion based Multi-head Attention), a lightweight alternative designed for efficiency. LAMA achieves state-of-the-art performance on text classification while maintaining a linear scaling factor and high parameter efficiency.
Methodology: The LAMA Architecture
LAMA streamlines the attention process by moving away from dense word-to-word querying. Instead, it employs a "global-to-local" approach to identify salient features within a document.
The core technical innovations include:
- Global Context Querying: Rather than every word attending to every other word, LAMA computes attention scores by querying a single global context vector (initialized as the mean of word embeddings). This reduces complexity from $O(n^2)$ to $O(n)$.
- Low-Rank Bilinear Pooling: To support multiple attention heads without a massive parameter explosion, the authors use a low-rank matrix factorization trick. The bilinear projection matrix is factorized into two low-rank matrices ($P$ and $Q$). A Hadamard product (element-wise multiplication) then produces multiple attention distributions efficiently.
- Bi-GRU Backbone: A Bi-directional Gated Recurrent Unit (bi-GRU) captures sequential dependencies before the attention layer is applied.
- Disagreement Regularization: An auxiliary loss term penalizes heads for being too similar, ensuring that different attention heads capture distinct "aspects" of the text.
Experimental Setup
The model was evaluated across several diverse text classification benchmarks, including:
- Yelp: Business ratings and sentiment analysis.
- IMDB: Binary sentiment analysis for movie reviews.
- News Aggregator & Reuters (r8): Topic classification for news articles.
Key Results and Efficiency
LAMA demonstrated performance comparable to or exceeding strong baselines like CNNs, Bi-GRUs, and standard Self-Attention Networks (SAN). Notably, LAMA outperformed BERT-base on several datasets while being significantly faster to train and having a smaller memory footprint.
Parameter and Computational Efficiency
LAMA is approximately three times more parameter-efficient than a standard Transformer Encoder. Because of the low-rank trick, increasing the number of attention heads only marginally impacts the total parameter count. In contrast, standard Transformers scale linearly with the number of heads. For classification tasks, performance generally peaked around 15 attention heads.
Interpretability
The global context query makes attention weights highly interpretable. The model effectively identifies key sentiment-bearing words (e.g., "amazing") and correctly adjusts their importance based on the surrounding context (e.g., "not amazing").
Conclusion and Future Work
LAMA proves that for many supervised tasks like classification, the full complexity of word-to-word attention is often redundant. By utilizing global-to-local querying and low-rank factorization, LAMA provides a path toward high-performance NLP on edge devices.
Future research aims to address the sequential bottleneck of the Bi-GRU backbone by exploring purely non-recurrent architectures with positional embeddings to regain the parallelization benefits of Transformers. Additionally, the authors suggest "phrase-level" queries as a potential middle ground to further enhance document representation.
Summary updated