In natural language processing, models often behave like impatient readers. Whether they are summarizing a scientific paper, retrieving a relevant document, or generating an embedding, they exhibit a systematic preference for information located at the beginning of a text. This phenomenon, known as lead bias or positional bias, prevents models from forming a holistic understanding of long-form content where critical insights are often distributed across the middle or end.
The Origin of the Lead Bias
The tendency of models to over-prioritize the "head" of a document is rarely a deliberate architectural choice; rather, it is an emergent property of how they are trained.
As Samarth Goel and colleagues point out in Quantifying Positional Biases in Text Embedding Models, the frequent truncation of long sequences during pre-training creates an exposure imbalance. Tokens at the start of a sequence appear in every training sample, while tokens at the end are frequently cut off, leading models to develop a structural preference for early context.
Furthermore, the datasets used for fine-tuning—particularly in news summarization—reinforce this behavior. News articles are traditionally written in an "inverted pyramid" style, front-loading the most important facts. Models trained on these corpora, such as those analyzed by Tian Shi in Neural Abstractive Text Summarization with Sequence-to-Sequence Models, learn that the first few sentences are the most reliable predictors of the summary, causing them to ignore salient information located deeper in the text.
The Middle Curse and the U-Shaped Curve
While older models suffered primarily from a "head" bias, modern Large Language Models (LLMs) with massive context windows exhibit a more complex "middle-curse." In On Context Utilization in Summarization with Large Language Models, Aixin Sun and Mathieu Ravaut demonstrate that LLMs tend to utilize information from the very beginning and the very end of a document, while neglecting the middle.
This U-shaped performance curve suggests that even as we expand context windows to 100k tokens or more, the model's "attention" is not uniformly distributed. This is particularly problematic for scientific papers or books, where the methodology and results—the core of the contribution—are buried in the middle sections.
Strategies for Global Context Capture
Researchers have proposed several architectural and procedural shifts to force models to "read" the entire document.
1. Divide and Conquer via Communicating Agents
One approach is to break the document into manageable chunks and process them in parallel. Asli Celikyilmaz and Yejin Choi introduced a system of Deep Communicating Agents where multiple encoder agents handle different sections of a document. By allowing these agents to exchange messages across layers, the model captures global context more effectively than a single-encoder system, ensuring that information from the middle and end is not lost.
2. Structural and Discourse Awareness
Another strategy involves making the model aware of the document's internal structure. In A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents, Arman Cohan and colleagues utilized hierarchical representations to attend to different sections explicitly. By treating a document as a collection of sections rather than a flat sequence of tokens, the model can distribute its "importance budget" across the introduction, methods, and conclusion. Cohan later extended this in On Generating Extended Summaries of Long Documents by using a multi-task learning objective that forces the model to predict which section a sentence belongs to, effectively anchoring the encoder's attention to the entire document length.
3. Non-Sequential Selection
Standard sequence-to-sequence models process text linearly, which naturally favors earlier tokens. Yue Dong proposed BanditSum, which reframes extractive summarization as a contextual bandit problem. By evaluating sentence importance across the entire document simultaneously rather than sequentially, the model avoids the "fake" extractive labels that often bake in lead bias. Similarly, the SimCLS framework by Yixin Liu uses contrastive learning to score candidate summaries against the entire source document, mitigating the inherent bias of the initial generation phase.
Moving Beyond News
A final piece of the puzzle is the data itself. Faisal Ladhak introduced WikiLingua, a benchmark based on procedural WikiHow articles. Unlike news, procedural text distributes salient information across sequential steps. Training on such structured data forces models to identify important information regardless of its position, providing a more robust foundation for summarizing complex, long-form content like books or technical manuals.