BOS Summary: The First Token Matters

Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Deng Pan, Arun Ross, Dongxiao ZhuarXiv 2026 · arXiv

TL;DR

BOS Summary is a training-free, inference-time intervention that repurposes the uninformative "attention sink" at the start of a sequence into a rich, one-token summary of the input. By relaxing the causal mask for just the beginning-of-sequence (BOS) token at a single transformer layer, the method significantly improves reasoning and grounding in both LLMs and vision-language models (VLMs) with zero additional parameters or decoding overhead.

What Problem Does This Paper Address?

Large Language Models (LLMs) and Vision-Language Models (VLMs) have a curious quirk: they often exhibit "attention sinks." The very first token in a sequence—the Beginning-of-Sequence (BOS) token—receives a massive, disproportionate amount of attention from all subsequent tokens.

Under standard causal masking, however, the BOS token can only attend to itself. This means that while later tokens are constantly "looking back" at the BOS token throughout a long generation, the representation they are accessing is essentially a "black hole" of information—it carries no specific data about the actual prompt or image. This wasted pathway contributes to "attention drift," where models lose track of the original instruction or visual evidence during long-form generation, leading to hallucinations and context loss.

Methods

The authors introduce BOS Summary, a "plug-and-play" method that exploits this existing attention pathway.

The core idea is simple: during the prefill stage (when the model first processes the prompt), the causal constraint is relaxed for the BOS query row alone at one selected transformer layer. This allows the BOS token to "see" and aggregate information from all visual and textual tokens in the prompt.

Figure 1. BOS Summary. Standard causal attention leaves BOS attending only to itself, making it an uninformative sink despite re- ceiving heavy attention from all later tokens (top). We unmask the BOS row at one selected layer so BOS attends to the visual and instruction tokens, turning it into a context aggregator that later tokens read from throughout generation (bottom). All other atten- tion patterns, the architecture are unchanged.

Key technical details include:

  • Single-Layer Intervention: The mask is only modified at one layer ($\ell^\star$). The resulting "summary" representation then propagates naturally through subsequent layers.
  • Zero Overhead: Because it only changes a few entries in an attention mask during prefill, it adds no parameters, auxiliary modules, or extra computation time during the autoregressive decoding phase.
  • Training-Free: It works "off-the-shelf" with existing pretrained models, though the authors also show it can provide further gains when used during fine-tuning.

Datasets

The authors evaluated BOS Summary across a wide array of benchmarks for both multimodal and text-only tasks:

  • Multimodal (VLMs): MMStar (general understanding), M3CoT (multi-step reasoning), POPE (object hallucination), and MMBench.
  • Text-Only (LLMs): RACE (reading comprehension), LogiQA (logical reasoning), MathQA (mathematical reasoning), and QuAC (conversational QA).
  • Models Tested: Qwen2.5-VL, Qwen3-VL, Gemma-3 (4B & 12B), and Llama-3.1-8B.

Key Insights and Ideas

  • The Sink is Not Just a No-Op: Contrary to some theories that sinks are just "placeholders" for unnecessary attention, this paper shows that the sink's content matters. When you give the sink meaningful content, the model uses it to maintain better grounding.
  • Input-Specificity Emerges: Mechanistic analysis reveals that BOS Summary makes the BOS token highly input-specific. Retrieval probes showed that the BOS representation at later layers could identify its source input with up to 97% accuracy.
  • Weakening the "Black Hole": Paradoxically, making the BOS token more informative actually reduces its dominance as an attention sink. By giving the BOS token actual content, later layers redistribute some of their attention back toward other content-bearing tokens in the prompt.
  • Architecture-Dependent Redistribution: While all models benefited, they reallocated "released" attention differently. Gemma models shifted attention toward image tokens, while Qwen models shifted it toward the text instructions.
Figure 2. BOS Summary makes the BOS representation input-dependent and input-specific. Results are shown for Qwen2.5-VL-7B- Instruct and Qwen3-VL-8B-Instruct on MMStar. (a) Dispersion measures how much the BOS representation varies across inputs. BOS Summary increases dispersion toward the level of ordinary prompt tokens. (b) Retrieval accuracy measures whether the BOS representation can identify its source input. BOS Summary performs well above chance, peaking at about 68% for Qwen2.5-VL-7B and 97% for Qwen3- VL-8B.

Key Results

  • Performance Gains: BOS Summary consistently outperformed direct prompting and Chain-of-Thought (CoT) baselines. On MMStar and M3CoT, it improved accuracy by 0.74–2.58 percentage points over the strongest baselines.
  • Superior to Other Sink Methods: It outperformed recent sink-intervention methods like SinkTrack and ZeroTuning on multi-step reasoning tasks.
  • Hallucination Reduction: Qualitative results showed the model was better at attending to small, distant details in images (like a person in the background) that standard CoT missed.
  • Fine-Tuning Synergy: When used during LoRA fine-tuning on Llama-3.1-8B, it provided an additional 3.07% accuracy boost on LogiQA compared to standard fine-tuning.
Figure 3. BOS Summary reduces attention to the BOS sink and redistributes it toward the prompt. Results are shown for five model– modality settings using 500 samples each. (a) Attention assigned to BOS across model depth for the unmodified model (gray) and BOS Summary (blue); each panel reports the mean change. (b) Change in attention share for BOS, Context (image or passage), and Question (question and instruction tokens), measured in percentage points. All measurements use a deterministic prefill pass with eager attention.

Limitations and Future Work

  • Layer Selection: The current method relies on a model-specific calibration to find the optimal layer ($\ell^\star$) for the intervention. The authors suggest that future work could develop more general, transferable rules for layer selection.
  • Optimization Dynamics: The paper focuses on inference-time behavior; how the BOS pathway changes during full-scale pretraining with this modified mask remains an open question.
  • Task Scope: While effective for reasoning and QA, the authors note that the benefits should be further explored in unrestricted, open-ended long-form generation.

Summary updated

Abstract

Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention.
BibTeX
@misc{zhu2026bos,
  author = {Hui Zhu and Saleh Zare Zade and Rafi Ibn Sultan and Deng Pan and Arun Ross and Dongxiao Zhu},
  title = {{BOS Summary: The First Token Matters}},
  year = {2026},
  eprint = {2611.23508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23508},
}
Figure