Introduction
The Attention mechanism has become the cornerstone of modern artificial intelligence, particularly within Transformer architectures. However, its computational cost scales quadratically ($O(N^2)$) with sequence length, posing significant challenges for long-context applications. While Linear Attention variants have reduced this complexity to $O(N)$, they often retain complex machinery. This paper introduces Relation, a simplified mechanism that questions whether pairwise token interactions are strictly necessary for effective context modeling. By utilizing a global context vector and element-wise operations, Relation achieves linear time complexity and superior computational efficiency while maintaining competitive performance on standard benchmarks.
Methodology: The Relation Mechanism
The core philosophy of Relation is to replace expensive $N \times N$ similarity matrices with a "global vibe" vector that weights local features. The process follows a streamlined four-step workflow:
- Dual Projection: The input matrix $X \in \mathbb{R}^{N \times m}$ is projected into two distinct representations, $G$ and $H$, using point-wise Multi-Layer Perceptrons (MLPs).
- Global Aggregation: An element-wise mean is calculated across all vectors in $H$ to produce a single global context vector $h'$.
- Context Injection: This global vector is broadcasted across the sequence and combined with $G$ via a Hadamard product (element-wise multiplication).
- Output Generation: A final linear projection and activation function yield the output matrix in $\mathbb{R}^{N \times d}$.
By avoiding matrix-matrix multiplications, Relation reduces the operational count significantly compared to both standard and Linear Attention. For a sequence of length $N$, Relation requires approximately $Nd^2 + 2Ndm + Nd$ operations, whereas Linear Attention requires $2Nd^2 + 3Ndm$.
Experimental Setup
The effectiveness of Relation was evaluated against standard Attention, Linear Attention, and a point-wise MLP baseline (no context) across two primary categories:
- General Language Understanding: The GLUE benchmark, comprising 11 diverse tasks such as CoLA, SST-2, and MNLI.
- Long-Sequence Efficiency: Datasets including IMDb, Reuters, and WikiText-103 (with sequences up to 6,231 tokens) were used to measure wall-clock computation time.
Results and Discussion
Performance on NLU Tasks
Experimental results on the GLUE benchmark demonstrate that Relation achieves performance parity with Attention-based models. As shown in the performance table, at a parameter size of 256, Relation reached an average score of 54.2, slightly outperforming standard Attention (51.9) and matching Linear Attention (54.1). Notably, the baseline MLP without context failed significantly on tasks requiring sequence-wide understanding (e.g., the AX diagnostic task), confirming that Relation's simple global mean effectively captures meaningful contextual information.
Computational Efficiency
The primary advantage of Relation is its speed. In long-sequence tasks like WikiText-103, Relation (137s per epoch) significantly outperformed Linear Attention (185s) and standard Attention (1,540s). This efficiency stems from the reduction of total operations and a smaller memory footprint, making it highly suitable for hardware with limited resources.
Conclusion
Relation demonstrates that for many natural language tasks, complex relational mapping can be effectively replaced by simple global feature gating. While its performance in extremely deep architectures or on tasks requiring fine-grained token dependencies remains a subject for future study, its $O(N)$ efficiency makes it a compelling candidate for processing exceptionally long sequences in domains like genomics or long-form document analysis.
Summary updated