Gold Panning: Turning Positional Bias into Signal for Multi-Document LLM Reasoning

Adam Byerly, Daniel Khashabi

Introduction

Large Language Models (LLMs) have revolutionized information retrieval and reasoning, yet they remain plagued by a phenomenon known as the "lost in the middle" effect. When processing long contexts, models typically exhibit a strong position bias, demonstrating high accuracy for information located at the beginning or end of a prompt while struggling with data buried in the center. Traditionally, this bias has been viewed as a failure mode to be mitigated through random shuffling or architectural changes.

In this paper, we present Gold Panning, a novel black-box framework that reframes positional bias as a valuable diagnostic signal. By modeling context positions as noisy detectors and employing Bayesian active search, Gold Panning identifies relevant information—the "needle in the haystack"—with significantly higher efficiency than existing ensembling techniques.

The Gold Panning Framework

Gold Panning operates as an iterative Bayesian search process. Unlike standard methods that treat all context positions as equally informative, Gold Panning explicitly leverages the model's known strengths and weaknesses across different positions.

The Gold Panning pipeline: from calibrating positional bias to iterative document reordering and belief updates.

The framework consists of four primary stages:

  1. Calibration: The system first establishes a "positional diagnosticity profile" for the specific LLM. By placing known relevant documents at various positions, it measures the True Positive Rate (TPR) and False Positive Rate (FPR) for each location, identifying which parts of the context are most reliable.
  2. Bayesian Beliefs: A probability distribution (belief) is maintained for every document in the collection, representing the likelihood that it contains the required information.
  3. Signal Anchoring (Strategic Shuffling): In each iteration, documents are reordered based on current beliefs. Documents with the highest belief scores are placed in the most "diagnostic" positions—those where the model is most accurate. This "anchoring" strategy ensures that promising candidates receive the highest quality attention.
  4. Iterative Updates: The model is queried, and its citations are used to update the Bayesian beliefs. This cycle repeats until a confidence threshold is reached.

Theoretically, this greedy approach allows the system to identify a target document among $N$ candidates in $O(\log N)$ rounds, providing a robust mathematical foundation for its efficiency.

Experimental Evaluation

The researchers evaluated Gold Panning using the FLenQA (Factuality in Long-context QA) benchmarks, specifically the MonoRel (monotonic transitive reasoning) and PIR (compositional reasoning) tasks. These were structured as Multi-Document Question Answering (MDQA) problems with up to 500 documents. Testing was conducted across the Gemma-3 (12B, 27B) and OLMo 3 (7B, 32B) model families.

Results and Discussion

The core insight of Gold Panning is that model biases are predictable regularities rather than random errors. By treating "sharp" zones of the context window as high-quality sensors, the framework achieves remarkable efficiency gains.

F1 Score vs. Iteration: Gold Panning (orange) reaches near-peak performance in half the iterations required by standard baselines (black).

Key findings include:

  • Efficiency: Gold Panning matches the accuracy of Permutation Self-Consistency (PSC) while requiring 30–65% fewer LLM queries. On Gemma-3-12B, near-perfect F1 scores were achieved in just 4 iterations, whereas the baseline required 8.
  • Anchoring vs. Exploration: The "GP-Belief" (anchoring) strategy consistently outperformed uncertainty-reduction strategies (GP-Entropy). Probing ambiguous documents often introduced noise, while anchoring likely signals accelerated the separation of relevant information from distractors.
  • Robustness: The method proved highly resilient to calibration errors. As long as the relative ranking of positions (best vs. worst) remained accurate, the performance gains persisted even if the exact TPR/FPR estimates were slightly off.

Conclusion and Future Directions

Gold Panning demonstrates that LLM limitations, such as positional bias, can be systematically exploited to improve system performance. By transforming a "bug" into a "feature," the framework provides a powerful tool for long-context reasoning.

While highly effective, the iterative nature of the process introduces latency compared to single-query methods. Future work may explore using internal model states to refine belief updates, potentially reducing the need for multiple full-text queries while maintaining the benefits of the Bayesian framework. Additionally, extending the "signal anchoring" dynamics to scenarios with multiple relevant documents remains a promising area for investigation.

Summary updated

Abstract

Large language models exhibit a strong position bias in multi-document contexts, systematically prioritizing information based on location rather than relevance. While existing approaches treat this bias as noise to be mitigated, we introduce GOLD PANNING BANDITS, a framework that leverages position bias as a diagnostic signal: by reordering documents and observing shifts in the model's responses, we can efficiently identify the most relevant content. We frame the problem of choosing reorderings as a bipartite matching problem. While an optimal assignment can be computed at each iteration with the Hungarian algorithm in $O(N^3)$ time, we propose a greedy $O(N \log N)$ strategy that achieves comparable performance by prioritizing the placement of the most uncertain documents in the most informative positions. Our approach identifies relevant documents using up to 65\% fewer language model queries than random permutation baselines on knowledge-intensive NLP tasks, substantially reducing computational cost without model retraining. This work demonstrates that inherent LLM biases can be transformed from liabilities into assets for efficient, inference-time optimization.
BibTeX
@misc{byerly2025gold,
  author = {Adam Byerly and Daniel Khashabi},
  title = {{Gold Panning: Turning Positional Bias into Signal for Multi-Document LLM Reasoning}},
  year = {2025},
  eprint = {2510.09770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.09770},
}
Figure