Abstract
The Integral (INTG) Transformer introduces a novel self-attention mechanism designed to "denoise" attention by averaging multiple signals sampled from the logit distribution prior to the softmax operation. This approach seeks a "Goldilocks" balance: reducing the disproportionate focus on uninformative tokens without completely eliminating their role as structural "attention sinks." Our empirical results demonstrate that the Integral Transformer outperforms Vanilla, Cog, and Differential Transformers across a suite of knowledge, reasoning, and long-context benchmarks.
Introduction
Standard Transformer architectures often suffer from "attention noise," where the self-attention mechanism allocates excessive scores to semantically uninformative tokens, such as punctuation or the beginning-of-sequence ([BOS]) token. While these tokens can serve as "attention sinks" that help manage context, over-concentration can limit the model's capacity for content-rich representations.
Previous attempts to mitigate this, such as the Differential Transformer, subtract attention maps to cancel out noise. However, such methods can be overly aggressive, often assigning negative weights to these tokens and potentially discarding useful structural information. The Integral Transformer addresses this by integrating multiple signals to improve the signal-to-noise ratio without "throwing the baby out with the bathwater."
Methodology: The Integral Approach
The Integral Transformer is inspired by "spatial antenna diversity" in communication systems, where multiple antennas receive the same signal to mitigate fading. Instead of subtracting signals, the Integral Transformer averages them.
For a single attention head, the model generates $S$ distinct pairs of Query ($Q$) and Key ($K$) projections. The dot-product logits ($QK^\top$) for all $S$ signals are averaged before the softmax operation. This logit averaging acts as a geometric mean in the probability space, providing greater resistance to outliers compared to averaging after the softmax. To maintain a constant parameter count relative to a vanilla Transformer, the head dimension is divided by $S$.
Furthermore, the authors implement Partial Depth Denoising. Analysis revealed that attention noise is more prevalent in upper layers. Consequently, the hybrid architecture utilizes vanilla softmax attention in the first 50% of layers and the Integral mechanism in the remaining 50%.
Experimental Setup
Models were pretrained at two scales:
- 125M parameters: Trained on 28B tokens from the Cosmopedia v2 corpus.
- 1.2B parameters: Utilizing a Llama-2 architecture, trained on 128B tokens from a mix of FineWeb-Edu and Cosmopedia v2.
Evaluation spanned eight standard benchmarks (Winogrande, ARC, HellaSwag, PIQA, OpenBookQA, BoolQ, and MMLU) and long-context capabilities via LongBench.
Results and Discussion
The Integral Transformer achieved an average score of 48.9% across the eight tasks at the 1.2B scale, surpassing both the Vanilla (47.2%) and Differential (47.6%) baselines. In long-context evaluations (2K to 8K tokens), it outperformed variants on 12 out of 16 datasets, showing particular strength in summarization.
The Sink Balance
A key finding is the model's handling of attention sinks. While the Differential Transformer often suppresses the [BOS] token to near-zero or negative weights, the Integral Transformer maintains a positive but reduced intensity.
This "denoised" intensity preserves the structural utility of the [BOS] token while freeing up attention bandwidth for content-carrying words. Additionally, the Integral Transformer maintains a higher effective rank in final layers, mitigating the "rank collapse" often seen in deep models and leading to more expressive representations.
Conclusion
The Integral Transformer provides a robust method for denoising self-attention by averaging logits across multiple projections. By striking a balance between noise reduction and structural preservation, it achieves superior performance across various scales and tasks. Future work will investigate the scaling properties of this mechanism beyond 1.2B parameters and explore more sophisticated sampling methods for the latent logit distribution.
Summary updated