Input Order Sensitivity in Language Models

917 concepts479 papers14 proposals

When we evaluate a Large Language Model (LLM) on a multiple-choice question or a list of options, we assume the model weighs the semantic content of each choice equally. However, a robust body of research reveals a fundamental flaw in this assumption: language models are profoundly sensitive to the order and positioning of their inputs. If you take a benchmark and simply shuffle the choices, a model's accuracy can plummet, or its final judgment can completely flip.

This phenomenon, known as positional bias or input order sensitivity, indicates that a model's output is often driven by the structural layout of the prompt rather than stable, content-driven reasoning. To build reliable systems, we must understand why models fall back on these positional shortcuts and how they distort our evaluations.

Primacy, Recency, and the Fallback Heuristic

Why do models care whether an option is labeled 'A' or 'C'? To build intuition, we must look at how transformers process sequences and how they handle uncertainty.

When presented with a list of choices or candidates, models frequently demonstrate "serial position effects"—cognitive shortcuts where they disproportionately favor information at the very beginning (primacy) or the very end (recency) of a sequence. As Soroush Vosoughi et al. demonstrate in Serial Position Effects of Large Language Models, these primacy and recency effects are pervasive across architectures, suggesting they are a fundamental characteristic of how transformers process sequential context rather than a simple prompting artifact. For instance, in information retrieval and listwise re-ranking tasks, Sunhao Dai et al. observe in Unifying Bias and Unfairness in Information Retrieval that models systematically favor items placed at the beginning or end of a sequence, often ignoring highly relevant information buried in the middle.

This positional preference becomes particularly acute under two conditions: high task complexity and high uncertainty. When a reasoning task exceeds a model's capacity, or when it cannot confidently determine the correct answer, it defaults to a structural fallback heuristic. Instead of performing deep semantic analysis, it "blindly guesses" based on position.

Olga Loginova et al. explore this in Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models, showing that selection bias acts as an architectural fallback mechanism when the cognitive load of a task becomes too high. Similarly, Zheqi He et al. find in CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning that models rely on these structural shortcuts primarily when they lack the domain knowledge to solve the problem.

Furthermore, the mechanism often operates on relative plausibility rather than absolute truth. As Haochun Wang et al. show in LLMs May Perform MCQA by Selecting the Least Incorrect Option, models solve multiple-choice tasks by executing a relative ranking of the options in the prompt, choosing the "least incorrect" option rather than identifying an independently correct fact. When the order of these options changes, the relative comparison dynamics shift, causing the model's internal heuristic to select a different answer.

The Illusion of Conceptual Understanding

The practical consequence of input order sensitivity is that standard evaluation metrics can severely overestimate a model's true capabilities. A model might achieve a high score on a benchmark simply because the correct answers happen to align with its internal positional preferences.

This illusion of competence spans multiple domains:

  • Spatial and Physical Reasoning: In Choosing 'Right' from Wrong, Giselle Zeno et al. demonstrate that Large Multimodal Models often favor specific answer positions (like option 'A') regardless of the visual content, meaning high benchmark scores can mask a total failure in actual spatial reasoning. This is mirrored in physical logic evaluations; Cory Paik et al. show in PROST: Physical Reasoning about Objects through Space and Time that models rely heavily on prompt structure and choice ordering rather than an internal representation of physical laws like gravity or mass.
  • Code Comprehension: Even in highly structured domains like software engineering, Dung Manh Nguyen et al. reveal in CodeMMLU that shifting the position of a correct answer drastically degrades accuracy, exposing a reliance on structural heuristics over true code execution reasoning.
  • Model-as-a-Judge Evaluations: When LLMs are used to evaluate other models, they exhibit a systematic preference for the first candidate solution presented. James Xu Zhao et al. document this in Automatic Model Selection with Large Language Models for Reasoning, noting that this positional bias severely complicates objective model selection.

When these non-semantic structural elements are perturbed, the fragility becomes obvious. Khanh-An C. Quan et al. show in Evaluating VQA Models' Consistency in the Scientific Domain that multimodal models frequently alter their answers under simple option permutations, proving that their apparent accuracy is heavily anchored to positional artifacts. Similarly, Lukasz Borchmann demonstrates in ARC 'Challenge' Is Not That Challenging that presenting multiple-choice options simultaneously triggers positional preferences that can skew accuracy metrics, masking the true difficulty of a benchmark.

De-biasing the Prompt Architecture

Understanding why order sensitivity happens has led to targeted mitigation strategies that move beyond naive prompt engineering. Because these biases are deeply embedded, simple textual warnings often fail; Koh Takeuchi et al. note in Cognitive Biases in Large Language Models that positional shortcuts remain highly resistant even when models are explicitly told to ignore them.

Instead, effective solutions alter the structural presentation or calibrate the output distribution directly:

  • Option Shuffling and Masking: In QMOS, Blessed Guda et al. combine a question-masked loss function with dynamic option shuffling during training and inference to break the statistical link between specific option labels and correctness.
  • Distributional Calibration: Rather than modifying the prompt, methods like the one proposed by Changho Shin et al. in OTTER use optimal transport at inference time to mathematically enforce label neutrality, aligning the model's raw prediction logits with a balanced target distribution.
  • Algorithmic Aggregation: Solutions can also target this at the system level. For instance, Sana Ebrahimi introduces REQUAL-LM, a black-box wrapper that uses Monte Carlo sampling to aggregate multiple permuted outputs into an equitable centroid, neutralizing the primacy effect through structural averaging.

By treating input order sensitivity not as a random error, but as a predictable architectural fallback mechanism, researchers can design evaluation frameworks that isolate genuine conceptual understanding from superficial structural alignment.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_positional_bias_input_order_sensitivity_8afbd082,
  title = {positional bias (input order sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/positional-bias-input-order-sensitivity/rdr_8afbd0826cc842af9148ea0d258660b9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_positional_bias_input_order_sensitivity_8afbd082_v1,
  title = {positional bias (input order sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/positional-bias-input-order-sensitivity/rdr_8afbd0826cc842af9148ea0d258660b9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake