ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training

Feijiang Han, Xiaodong Yu, Jianheng Tang, Qingyun Zeng, Licheng Guo, Lyle Ungar

Abstract

ZeroTuning is a training-free, inference-time method that significantly improves Large Language Model (LLM) performance by scaling the attention weights of the initial token (e.g., <BOS>). By treating this "attention sink" as a universal control lever, ZeroTuning reshapes a model's focus, reduces uncertainty, and corrects systematic biases without parameter updates or complex heuristics. Evaluated across 15 datasets, it demonstrates substantial gains, such as a 19.9% relative improvement in text classification for Llama-3.1-8B.

Introduction

While Large Language Models (LLMs) are powerful, they often require "steering" at inference time to optimize performance on specific tasks. A common approach is token-level attention tuning, where researchers manually or heuristically identify "important" tokens (like keywords in a query) and boost their attention scores. However, this approach is often fragile, relying on task-specific heuristics that can introduce bias, and is incompatible with modern, high-performance inference kernels like FlashAttention which act as "black boxes."

ZeroTuning addresses these issues by intervening on a single, universal token: the initial "Beginning of Sentence" (BOS) token. This token naturally acts as an "attention sink" in modern Transformers, serving as a high-leverage control point for the entire model's attention mechanism.

Methods

ZeroTuning applies a scaling factor ($\gamma$) to the attention logits of the initial token. In modern Transformers, this token naturally acts as an "attention sink"—a place where the model "parks" extra attention when it doesn't find other relevant tokens to attend to. By modifying this sink, researchers can reshape the global attention distribution. Up-scaling ($\gamma > 1$) increases the sink effect, which theoretically flattens the attention distribution across the rest of the sentence, promoting a more "holistic" read. Conversely, down-scaling ($\gamma < 1$) decreases the sink effect, sharpening the model's focus on specific, salient spans of text.

The method is kernel-agnostic, modifying Query or Key states of the initial token rather than the full attention matrix, making it compatible with FlashAttention and requiring only four lines of code change.

ZeroTuning implementation and performance summary.

The optimal scaling can be found via two modes. The Supervised Mode uses a small labeled validation set (e.g., 500 examples) to find the best scaling factor and identify which specific attention heads respond best to tuning. The Unsupervised Mode is a "plug-and-play" version that finds the optimal scaling by minimizing the entropy (uncertainty) of the model's next-token predictions on unlabeled data.

Key Insights

A critical discovery of this research is the strong inverse correlation between prediction entropy and task accuracy. As shown in the experiments, the scaling factor that results in the lowest prediction entropy almost always results in the highest task accuracy. This discovery validates the unsupervised version of the tool, allowing for effective tuning without labeled data.

Correlation between scaling factor, accuracy, and entropy on SST-2 and LogiQA.

Furthermore, the effect of ZeroTuning is not uniform across the model. Tuning is most effective in early and middle layers, which handle representation learning. The method also helps models escape "format traps"—situations where a model provides an answer in the wrong format because it is fixated on local cues.

Key Results

ZeroTuning consistently outperformed vanilla inference and more complex prior methods like PASTA and ACT across 15 diverse datasets. For Llama-3.1-8B, it achieved relative improvements of 19.9% on classification and 4.5% on QA, while also increasing its MT-Bench score. The intervention adds negligible overhead to inference and works out-of-the-box with 4-bit and 8-bit quantization. Gains persisted even as context length increased and across different decoding strategies like Top-p or Beam Search.

Limitations and Future Work

ZeroTuning "unlocks" latent knowledge but cannot fix "confidently wrong" errors or fundamental factual deficiencies. Extreme scaling can lead to model degeneration, such as repetitive loops. Future work includes optimizing the search for $\gamma$ via Bayesian methods and exploring more granular, joint optimization of every attention head simultaneously.

Summary updated

Abstract

Training-free methods for enhancing large language models (LLMs) have attracted growing interest recently, with token-level attention tuning emerging as an interpretable and promising direction. However, existing methods typically rely on auxiliary mechanisms to identify important or irrelevant task-specific tokens, introducing potential bias and limiting applicability. In this work, we uncover a surprising and elegant alternative: the semantically empty initial token (e.g., <BOS> in Llama) serves as a powerful and underexplored control point for optimizing model behavior. Through theoretical analysis, we show that tuning the initial token’s attention sharpens or flattens the attention distribution over subsequent tokens, and its role as an attention sink amplifies this effect. Empirically, we find that: (1) tuning its attention improves LLM performance across tasks more effectively than tuning other task-specific tokens; (2) the effect follows a consistent trend across layers, with earlier layers having greater impact, but varies across attention heads, with different heads showing distinct preferences in how they attend to this token. Based on these findings, we propose ZeroTuning, a training-free approach that improves LLM performance by applying head-specific attention adjustments to this special token. Despite tuning only one token, ZeroTuning achieves higher average performance on text classification, multiple-choice QA, and multi-turn conversation tasks across models such as LLama, Qwen, and DeepSeek. For example, ZeroTuning improves Llama-3.1-8B by 11.71% on classification tasks, 2.64% on QA tasks, and raises its multi-turn score from 7.804 to 7.966. The method is also robust to limited resources, few-shot settings, long contexts, quantization, decoding strategies, and prompt variations. Our work sheds light on a previously overlooked control point in LLMs, offering new insights into both inference-time tuning and model interpretability.
BibTeX
@inproceedings{han2026zerotuning,
  author = {Feijiang Han and Xiaodong Yu and Jianheng Tang and Qingyun Zeng and Licheng Guo and Lyle Ungar},
  title = {{ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=EdkQ14FiiO},
}
Figure