TL;DR
BOS Summary is a training-free, inference-time intervention that repurposes the uninformative "attention sink" at the start of a sequence into a rich, one-token summary of the input. By relaxing the causal mask for just the beginning-of-sequence (BOS) token at a single transformer layer, the method significantly improves reasoning and grounding in both LLMs and vision-language models (VLMs) with zero additional parameters or decoding overhead.
What Problem Does This Paper Address?
Large Language Models (LLMs) and Vision-Language Models (VLMs) have a curious quirk: they often exhibit "attention sinks." The very first token in a sequence—the Beginning-of-Sequence (BOS) token—receives a massive, disproportionate amount of attention from all subsequent tokens.
Under standard causal masking, however, the BOS token can only attend to itself. This means that while later tokens are constantly "looking back" at the BOS token throughout a long generation, the representation they are accessing is essentially a "black hole" of information—it carries no specific data about the actual prompt or image. This wasted pathway contributes to "attention drift," where models lose track of the original instruction or visual evidence during long-form generation, leading to hallucinations and context loss.
Methods
The authors introduce BOS Summary, a "plug-and-play" method that exploits this existing attention pathway.
The core idea is simple: during the prefill stage (when the model first processes the prompt), the causal constraint is relaxed for the BOS query row alone at one selected transformer layer. This allows the BOS token to "see" and aggregate information from all visual and textual tokens in the prompt.
Key technical details include:
- Single-Layer Intervention: The mask is only modified at one layer ($\ell^\star$). The resulting "summary" representation then propagates naturally through subsequent layers.
- Zero Overhead: Because it only changes a few entries in an attention mask during prefill, it adds no parameters, auxiliary modules, or extra computation time during the autoregressive decoding phase.
- Training-Free: It works "off-the-shelf" with existing pretrained models, though the authors also show it can provide further gains when used during fine-tuning.
Datasets
The authors evaluated BOS Summary across a wide array of benchmarks for both multimodal and text-only tasks:
- Multimodal (VLMs): MMStar (general understanding), M3CoT (multi-step reasoning), POPE (object hallucination), and MMBench.
- Text-Only (LLMs): RACE (reading comprehension), LogiQA (logical reasoning), MathQA (mathematical reasoning), and QuAC (conversational QA).
- Models Tested: Qwen2.5-VL, Qwen3-VL, Gemma-3 (4B & 12B), and Llama-3.1-8B.
Key Insights and Ideas
- The Sink is Not Just a No-Op: Contrary to some theories that sinks are just "placeholders" for unnecessary attention, this paper shows that the sink's content matters. When you give the sink meaningful content, the model uses it to maintain better grounding.
- Input-Specificity Emerges: Mechanistic analysis reveals that BOS Summary makes the BOS token highly input-specific. Retrieval probes showed that the BOS representation at later layers could identify its source input with up to 97% accuracy.
- Weakening the "Black Hole": Paradoxically, making the BOS token more informative actually reduces its dominance as an attention sink. By giving the BOS token actual content, later layers redistribute some of their attention back toward other content-bearing tokens in the prompt.
- Architecture-Dependent Redistribution: While all models benefited, they reallocated "released" attention differently. Gemma models shifted attention toward image tokens, while Qwen models shifted it toward the text instructions.
Key Results
- Performance Gains: BOS Summary consistently outperformed direct prompting and Chain-of-Thought (CoT) baselines. On MMStar and M3CoT, it improved accuracy by 0.74–2.58 percentage points over the strongest baselines.
- Superior to Other Sink Methods: It outperformed recent sink-intervention methods like SinkTrack and ZeroTuning on multi-step reasoning tasks.
- Hallucination Reduction: Qualitative results showed the model was better at attending to small, distant details in images (like a person in the background) that standard CoT missed.
- Fine-Tuning Synergy: When used during LoRA fine-tuning on Llama-3.1-8B, it provided an additional 3.07% accuracy boost on LogiQA compared to standard fine-tuning.
Limitations and Future Work
- Layer Selection: The current method relies on a model-specific calibration to find the optimal layer ($\ell^\star$) for the intervention. The authors suggest that future work could develop more general, transferable rules for layer selection.
- Optimization Dynamics: The paper focuses on inference-time behavior; how the BOS pathway changes during full-scale pretraining with this modified mask remains an open question.
- Task Scope: While effective for reasoning and QA, the authors note that the benefits should be further explored in unrestricted, open-ended long-form generation.
Summary updated