Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration

Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, Yingyan Celine Lin

Abstract

Attention is a fundamental component behind the remarkable achievements of large language models (LLMs). However, our current understanding of the attention mechanism, especially regarding how attention distributions are established, remains limited. Inspired by recent studies that explore the presence of attention sinks in the initial token—which receives disproportionately large attention scores despite lacking semantic importance—this work delves deeper into this phenomenon. We provide a more profound understanding of the existence of attention sinks within LLMs and uncover ways to enhance accuracy by directly optimizing attention distributions without weight finetuning. Specifically, we discover that attention sinks occur not only at the start of sequences but also within later tokens, and that not all attention sinks have a positive impact on accuracy. Building upon these findings, we propose a training-free Attention Calibration Technique (ACT) that automatically optimizes attention distributions on the fly during inference. Extensive experiments validate that ACT consistently enhances the accuracy of various LLMs across different applications, achieving an average improvement of up to 7.30% in accuracy when applied to Llama-30B.

Introduction

The attention mechanism serves as the engine of the Transformer architecture, allowing models to weigh the importance of different tokens in a sequence. Recent research, such as StreamLLM, identified a "sink" effect where the very first token in a prompt often receives a massive, disproportionate amount of attention, even if it is a semantically neutral start-of-sentence marker. While this initial sink helps maintain model stability, it raises critical questions about the efficiency of attention allocation throughout the rest of the sequence.

This research, presented at ICML 2024, investigates whether these sinks are limited to the sequence start, whether they always benefit the model, and if manipulating them can improve performance without the need for expensive retraining or finetuning.

The Discovery of Hidden Sinks

By visualizing the attention maps of models like Llama-2 and Mistral during inference, we discovered that attention sinks are more pervasive than previously thought. These "hidden sinks" frequently emerge mid-sequence, often clustering around tokens with low semantic value, such as punctuation marks (":", ".") or newline characters (\n).

Our analysis revealed a "Wasted Attention" hypothesis: while the initial token sink acts as a necessary anchor for the softmax distribution, later sinks often act as a "dumping ground" for attention scores. These mid-sequence sinks can actively harm performance by "stealing" attention mass from semantically rich words that are crucial for reasoning and context understanding. Furthermore, we observed that these patterns are layer-dependent; while the first few layers show uniform attention, the middle layers develop heavy sinks, and the final layers transition toward local, diagonal patterns.

The Attention Calibration Technique (ACT)

To address these inefficiencies, we developed the Attention Calibration Technique (ACT). ACT is a lightweight, training-free mathematical adjustment made during the model's forward pass. The technique operates in two distinct phases:

  1. Offline Head Filtering: Not every attention head in a model benefits from calibration. We use a small held-out dataset to identify specific heads where suppressing sinks leads to improved performance.
  2. Inference-time Calibration: For the identified heads, the model automatically performs two steps during the forward pass:
  3. Identification: It flags "sink" tokens that receive more than a specific threshold ($\alpha$) of the average attention (typically $5\times$).
  4. Redistribution: It reduces these sink scores by a factor ($\beta$) and redistributes that "saved" attention mass back to the other tokens in the sequence. This ensures the mathematical requirements of the attention mechanism are maintained while focusing the model on meaningful content.

Experimental Results

We evaluated ACT across 18 datasets, including domain-specific multiple-choice tasks (Hellaswag, MMLU), text classification (SST2, AGNews), and open-ended QA (SQuAD, MT-Bench). The technique was tested on several popular open-source architectures, including Llama-2 (7B, 13B, 30B), Mistral-7B, and GPT-J-6B.

Key Performance Metrics

  • Accuracy Gains: ACT achieved an average accuracy improvement of up to 7.30% on Llama-30B.
  • Task-Specific Success: On the Hellaswag benchmark, improvements reached as high as 13.26%.
  • Conversational AI: In multi-turn conversations (MT-Bench), ACT improved Llama-2’s score by 0.13 points, a significant margin for a non-invasive, training-free intervention.
  • Robustness: The benefits of ACT remained consistent across 0-shot, 1-shot, 3-shot, and 5-shot settings, demonstrating its versatility across different prompting strategies.

Conclusion and Future Work

ACT demonstrates that LLM performance can be significantly enhanced by simply recalibrating how models "pay attention" to specific tokens. By identifying and mitigating the impact of hidden attention sinks in the middle layers of Transformers, we can recover "wasted" attention and redirect it toward semantically important information.

While ACT requires minimal hyperparameter tuning and a one-time offline filtering step, it offers a highly efficient path to boosting model accuracy without the computational burden of finetuning. Future research will explore the behavior of these sinks in even larger models (100B+ parameters) and alternative architectures like Mixture-of-Experts (MoE) to determine if these patterns scale or evolve in different structural contexts.

Summary updated

Abstract

Attention is a fundamental component behind the remarkable achievements of large language models (LLMs). However, our current understanding of the attention mechanism, especially regarding how attention distributions are established, remains limited. Inspired by recent studies that explore the presence of attention sink in the initial token, which receives disproportionately large attention scores despite their lack of semantic importance, this work delves deeper into this phenomenon. We aim to provide a more profound understanding of the existence of attention sinks within LLMs and to uncover ways to enhance the achievable accuracy of LLMs by directly optimizing the attention distributions, without the need for weight finetuning. Specifically, this work begins with comprehensive visualizations of the attention distributions in LLMs during inference across various inputs and tasks. Based on these visualizations, to the best of our knowledge, we are the first to discover that (1) attention sinks occur not only at the start of sequences but also within later tokens of the input, and (2) not all attention sinks have a positive impact on the achievable accuracy of LLMs. Building upon our findings, we propose a training-free Attention Calibration Technique (ACT) that automatically optimizes the attention distributions on the fly during inference in an input-adaptive manner. Extensive experiments validate that ACT consistently enhances the accuracy of various LLMs across different applications. Specifically, ACT achieves an average improvement of up to 7.30% in accuracy across different datasets when applied to Llama-30B. Our code is available at https://github.com/GATECH-EIC/ACT.
BibTeX
@inproceedings{yu2024unveiling,
  author = {Zhongzhi Yu and Zheng Wang and Yonggan Fu and Huihong Shi and Khalid Shaikh and Yingyan Celine Lin},
  title = {{Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=DLTjFFiuUJ},
}
Figure