Abstract
Large Language Models (LLMs) utilizing Rotary Position Embeddings (RoPE) exhibit "attention waveforms"—periodic fluctuations in attention intensity that create blind spots within their context windows. This paper introduces Attention Buckets, a training-free inference method that employs parallel processes with diverse position embedding bases. By ensuring that information missed by one stream's "trough" is captured by another's "peak," Attention Buckets significantly enhances performance in complex tool-use and retrieval tasks, even surpassing GPT-4 on key benchmarks.
Introduction
As Large Language Models (LLMs) are increasingly deployed for long-context applications such as multi-step tool use and complex document retrieval, a critical assumption is made: that these models maintain uniform attention across their entire context window. However, recent findings reveal a significant vulnerability in models using Rotary Position Embeddings (RoPE), the standard for architectures like Llama and Qwen. These models exhibit a "waveform" pattern in their attention allocation, where the ability to focus on a token fluctuates based on its relative position. When crucial information falls into an attention "trough," the model often fails to retrieve it, leading to task failure.
The Attention Waveform Phenomenon
The authors characterize this issue using the "shortest stave" analogy: like a wooden bucket where the water level is limited by the shortest plank, an LLM's context awareness is limited by these positional troughs. As shown in Figure 1, accuracy in key-value retrieval tasks fluctuates dramatically depending on the target's index. This is not a content-based failure but a positional one, directly tied to the mathematical properties of RoPE.
Methodology: Attention Buckets
To mitigate these blind spots, the authors propose Attention Buckets, a training-free inference-time solution that runs multiple parallel streams with complementary attention patterns. The method consists of four primary components:
- Parallel Inference: The input is processed through $N$ parallel streams simultaneously.
- Diverse RoPE Bases: Each stream utilizes a unique base angle ($B$) for its RoPE. Since the base determines the frequency of the attention wave, varying $B$ shifts the locations of peaks and troughs.
- Complementary Waveforms: A search algorithm identifies a set of bases that ensure every position in the context is covered by at least one attention peak across the streams.
- Confidence-Weighted Aggregation: At each decoding step, the model calculates a confidence score for each stream's output. The final token is predicted using a weighted sum of these distributions, naturally favoring the stream that has "seen" the relevant information most clearly.
Experimental Evaluation
The method was rigorously tested across several demanding benchmarks:
- Tool-Use: ToolBench (16,000+ APIs) and ToolAlpaca (multi-agent simulation).
- Retrieval: NaturalQuestion (NQ) and WebQA for Open-Domain Question Answering and RAG.
- Synthetic Tasks: Key-Value retrieval to isolate positional effects.
Key Results
- Surpassing GPT-4: On the ToolBench benchmark, a 7B Llama-based model (ToolLlama) equipped with Attention Buckets achieved a 71.3% pass rate, exceeding GPT-4's performance (71.1%).
- Robust RAG Improvements: In ODQA tasks, the method provided stable gains of 1.4 to 1.8 percentage points over standard Llama-2-7B, outperforming existing context-optimization techniques like Attention Sorting.
- Generalizability: Improvements were consistent across different model scales (7B and 13B) and diverse task types.
Discussion and Conclusion
Attention Buckets offers a powerful, training-free mechanism to enhance the context awareness of LLMs. By leveraging the mathematical periodicity of RoPE, it transforms a structural weakness into a robust parallel processing system. While the method introduces computational overhead due to parallel streams, its high parallelizability on modern GPUs makes it a viable solution for high-stakes tool-use and retrieval applications. Future work will explore extending this logic to non-RoPE embedding schemes and optimizing the trade-off between the number of "buckets" and inference latency.
Summary updated