ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training

Feijiang Han, Xiaodong Yu, Jianheng Tang, Delip Rao, Weihua Du, Lyle Ungar

TL;DR

ZeroTuning is a training-free, inference-time method that boosts LLM performance by adjusting the attention weight of just a single token: the initial token (e.g., <BOS>). By leveraging this token's natural role as an "attention sink," ZeroTuning reshapes downstream attention distributions using only four lines of code, outperforming much more complex, task-specific attention steering techniques.

What Problem Does This Paper Address?

Inference-time attention tuning—methods that alter an LLM's attention distributions without updating its weights—has become a popular way to guide model behavior. Existing techniques like PASTA or ACT try to identify and up-weight "important" tokens specific to a given prompt.

However, this approach has two major flaws:

  1. Heuristic Dependency: Finding "important" tokens requires fragile heuristics or auxiliary LLMs, which can introduce bias or fail when token importance is ambiguous.
  2. Kernel Incompatibility: Modern, highly optimized attention implementations (like FlashAttention or SDPA) fuse operations in hardware, making explicit attention maps inaccessible for modification during the forward pass.

ZeroTuning bypasses these issues entirely by asking a radical question: Can we improve model performance by tuning a universal, task-agnostic token? The authors target the very first token of the sequence, turning a well-known architectural artifact into a powerful control knob.

Methods

Instead of hunting for task-specific keywords, ZeroTuning applies a head-specific scaling factor ($\gamma$) to the attention logits of the initial token during the forward pass.

Figure 1: Overview of the ZeroTuning method and its effectiveness. Left: Our method requires only a few lines of code to scale the initial token’s attention within the model’s forward pass. Right: This simple intervention yields significant and consistent improvements across a variety of LLMs and tasks.

The authors propose two ways to calibrate this scaling factor:

  • Supervised Calibration: Uses a small labeled validation set (e.g., 500 samples) to profile attention heads, classifying them as either "up-effective" (performance improves when initial token attention is amplified) or "down-effective." It then searches for the optimal $\gamma$ for the dominant head type.
  • Unsupervised Calibration: Eliminates the need for labeled data entirely. It searches for the optimal $\gamma$ by directly minimizing the average next-token prediction entropy over a batch of unlabeled inputs.

Crucially, because modifying attention maps directly is incompatible with FlashAttention, ZeroTuning achieves the exact same mathematical effect by scaling the underlying query or key states of the initial token, making it entirely kernel-agnostic.

Datasets

The method was evaluated across 15 diverse datasets divided into three main categories:

  1. Text Classification & Reasoning: SST-2, SST-5, MR, BoolQ, CB, TREC, and SUBJ.
  2. Domain-Specific Multiple-Choice QA: MMLU, AQUA, MathQA, LogiQA, CQA, PIQA, and ARCC.
  3. Multi-Round Conversation: MT-Bench.

Evaluations were performed using several prominent open-weight models, including Llama-3.1-8B-Instruct, Llama-2-13B-Chat, Qwen-2-7B, and DeepSeek-R1-14B.

Key Insights and Ideas

  • The Initial Token as a Lever: Because the initial token naturally functions as an "attention sink" (drawing massive attention weights regardless of semantic content), it acts as a mathematical lever. Small changes to its attention logits compress or expand the relative attention differences among all subsequent tokens.
  • Global vs. Local Focus: Up-scaling the initial token's attention ($\gamma > 1$) flattens the remaining distribution, forcing the model to integrate context more globally (highly effective for sentiment tasks trapped by local negative words). Down-scaling ($\gamma < 1$) sharpens the remaining distribution, helping the model focus tightly on critical local evidence (ideal for long-context QA).
  • The Entropy-Accuracy Mirror: The authors discovered a striking inverse correlation between next-token prediction entropy and final task accuracy. Minimizing output uncertainty perfectly aligns with maximizing performance, providing a robust foundation for their unsupervised mode.
Figure 3: A strong inverse correlation between accuracy (orange) and next-token prediction entropy (blue).

Key Results

ZeroTuning consistently outperformed vanilla inference and prior attention-steering baselines across models and tasks:

  • Llama-3.1-8B-Instruct: Achieved a relative improvement of 19.9% on text classification, 4.5% on multiple-choice QA, and raised its MT-Bench conversation score from 7.804 to 7.966.
  • Qwen-2-7B: Saw a massive 23.8% relative boost in text classification.
  • DeepSeek-R1-14B: Gained 6.2% on classification and 3.6% on multiple-choice tasks.
  • Robustness: The unsupervised entropy-minimization variant performed nearly on par with the supervised version. Furthermore, ZeroTuning maintained its performance gains even when models were heavily quantized (4-bit and 8-bit) or when long, distracting contexts were introduced.

Limitations and Future Work

  • Unlocking vs. Rewriting Knowledge: ZeroTuning excels at resolving low-confidence, ambiguous errors by refining the model's focus, but it cannot overwrite high-confidence factual errors rooted in flawed pretraining. It is a tool for unlocking latent knowledge, not a replacement for fine-tuning.
  • Extreme Tuning Risks: Pushing $\gamma$ to extremes causes degradation. Over-suppressing the initial token leads to repetitive generation loops, while over-amplifying it causes the model to ignore task instructions.
  • Calibration Overhead: While inference-time overhead is negligible, the offline search for $\gamma$ takes 4–5 minutes on a single GPU. The authors suggest that future work could explore more efficient search strategies, like Bayesian optimization, to lower this calibration cost.

Summary updated

Abstract

Token-level attention tuning, a class of training-free methods including Post-hoc Attention Steering (PASTA) and Attention Calibration (ACT), has emerged as a promising approach for improving frozen LLMs via interpretable interventions. However, these methods rely on auxiliary heuristics to identify important task-specific tokens, which can introduce bias and limit applicability when token importance is ambiguous or when optimized kernels make attention maps inaccessible. We propose a simpler alternative: intervening only on the initial token (e.g., BOS in LLaMA). We theoretically show that adding lightweight biases to this token's attention logits systematically shifts and reshapes downstream attention patterns - an effect amplified by its natural role as an attention sink. Empirically, we find that this tuning can improve LLM performance and better elicit pretrained knowledge, with stronger effects in early layers and distinct scaling preferences across attention heads. Building on these findings, we introduce ZeroTuning, a training-free method that improves LLM performance by applying head-specific attention adjustments to the initial token, requiring no parameter updates. We present two variants: a supervised mode that calibrates on validation examples, and an unsupervised mode that directly minimizes output entropy. ZeroTuning requires no KV-cache or decoding changes and is kernel-agnostic (works with SDPA and FlashAttention). It requires only four lines of modification to the standard LlamaAttention code, achieves gains across 15 datasets, and outperforms prior, more complex methods. For example, on Llama-3.1-8B, it yields relative improvements of 19.9% on classification, 4.5% on question answering, and 2.1% on dialogue. ZeroTuning also works out of the box with quantized inference and maintains its improvements as context length increases.
BibTeX
@inproceedings{han2026zerotuning,
  author = {Feijiang Han and Xiaodong Yu and Jianheng Tang and Delip Rao and Weihua Du and Lyle Ungar},
  title = {{ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=EdkQ14FiiO},
}
Figure