Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use

Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, Rui Yan

Abstract

Large Language Models (LLMs) utilizing Rotary Position Embeddings (RoPE) exhibit "attention waveforms"—periodic fluctuations in attention intensity that create blind spots within their context windows. This paper introduces Attention Buckets, a training-free inference method that employs parallel processes with diverse position embedding bases. By ensuring that information missed by one stream's "trough" is captured by another's "peak," Attention Buckets significantly enhances performance in complex tool-use and retrieval tasks, even surpassing GPT-4 on key benchmarks.

Introduction

As Large Language Models (LLMs) are increasingly deployed for long-context applications such as multi-step tool use and complex document retrieval, a critical assumption is made: that these models maintain uniform attention across their entire context window. However, recent findings reveal a significant vulnerability in models using Rotary Position Embeddings (RoPE), the standard for architectures like Llama and Qwen. These models exhibit a "waveform" pattern in their attention allocation, where the ability to focus on a token fluctuates based on its relative position. When crucial information falls into an attention "trough," the model often fails to retrieve it, leading to task failure.

Figure 1: The attention waveform phenomenon. (a) shows a retrieval task, (b) illustrates the resulting accuracy fluctuations, and (c) visualizes the periodic attention scores.

The Attention Waveform Phenomenon

The authors characterize this issue using the "shortest stave" analogy: like a wooden bucket where the water level is limited by the shortest plank, an LLM's context awareness is limited by these positional troughs. As shown in Figure 1, accuracy in key-value retrieval tasks fluctuates dramatically depending on the target's index. This is not a content-based failure but a positional one, directly tied to the mathematical properties of RoPE.

Methodology: Attention Buckets

To mitigate these blind spots, the authors propose Attention Buckets, a training-free inference-time solution that runs multiple parallel streams with complementary attention patterns. The method consists of four primary components:

  1. Parallel Inference: The input is processed through $N$ parallel streams simultaneously.
  2. Diverse RoPE Bases: Each stream utilizes a unique base angle ($B$) for its RoPE. Since the base determines the frequency of the attention wave, varying $B$ shifts the locations of peaks and troughs.
  3. Complementary Waveforms: A search algorithm identifies a set of bases that ensure every position in the context is covered by at least one attention peak across the streams.
  4. Confidence-Weighted Aggregation: At each decoding step, the model calculates a confidence score for each stream's output. The final token is predicted using a weighted sum of these distributions, naturally favoring the stream that has "seen" the relevant information most clearly.
Figure 2: The Attention Buckets architecture. Multiple parallel streams with different RoPE bases generate output distributions that are weighted and summed.

Experimental Evaluation

The method was rigorously tested across several demanding benchmarks:

  • Tool-Use: ToolBench (16,000+ APIs) and ToolAlpaca (multi-agent simulation).
  • Retrieval: NaturalQuestion (NQ) and WebQA for Open-Domain Question Answering and RAG.
  • Synthetic Tasks: Key-Value retrieval to isolate positional effects.

Key Results

  • Surpassing GPT-4: On the ToolBench benchmark, a 7B Llama-based model (ToolLlama) equipped with Attention Buckets achieved a 71.3% pass rate, exceeding GPT-4's performance (71.1%).
  • Robust RAG Improvements: In ODQA tasks, the method provided stable gains of 1.4 to 1.8 percentage points over standard Llama-2-7B, outperforming existing context-optimization techniques like Attention Sorting.
  • Generalizability: Improvements were consistent across different model scales (7B and 13B) and diverse task types.

Discussion and Conclusion

Attention Buckets offers a powerful, training-free mechanism to enhance the context awareness of LLMs. By leveraging the mathematical periodicity of RoPE, it transforms a structural weakness into a robust parallel processing system. While the method introduces computational overhead due to parallel streams, its high parallelizability on modern GPUs makes it a viable solution for high-stakes tool-use and retrieval applications. Future work will explore extending this logic to non-RoPE embedding schemes and optimizing the trade-off between the number of "buckets" and inference latency.

Summary updated

Abstract

In this paper, we demonstrate that an inherent waveform pattern in the attention allocation of large language models (LLMs) significantly affects their performance in tasks demanding a high degree of context awareness, such as utilizing LLMs for tool-use. Specifically, the crucial information in the context will be potentially overlooked by model when it is positioned in the trough zone of the attention waveform, leading to decreased performance. To address this issue, we propose a novel inference method named Attention Buckets. It allows LLMs to process their input through multiple parallel processes. Each process utilizes a distinct base angle for the rotary position embedding, thereby creating a unique attention waveform. By compensating an attention trough of a particular process with an attention peak of another process, our approach enhances LLM’s awareness to various contextual positions, thus mitigating the risk of overlooking crucial information. In the largest tool-use benchmark, our method elevates a 7B model to achieve state-of-the-art performance, comparable to that of GPT-4. On other benchmarks and some RAG tasks, which also demand a thorough understanding of contextual content, Attention Buckets also exhibited notable enhancements in performance.
BibTeX
@inproceedings{chen2024fortify,
  author = {Yuhan Chen and Ang Lv and Ting-En Lin and Changyu Chen and Yuchuan Wu and Fei Huang and Yongbin Li and Rui Yan},
  title = {{Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.601},
}
Figure