Order Sensitivity in Large Language Model Reasoning

50 concepts33 papers2 proposals

In formal logic, the validity of an argument is invariant to the order of its premises. If premise $A$ and premise $B$ together imply conclusion $C$, the deduction holds true whether you state $A$ first or $B$ first. However, for transformer-based language models, this mathematical symmetry does not exist. Shuffling the order of input facts can cause catastrophic performance drops, revealing that these models rely heavily on the linear presentation of information rather than a robust internal representation of logical relationships.

This phenomenon, known as premise order sensitivity, spans across various reasoning tasks. For instance, Xinyun Chen et al. demonstrated in Premise Order Matters in Reasoning with Large Language Models that simply permuting the order of input facts can degrade reasoning accuracy by over 30%. This brittleness is not confined to complex multi-step proofs; it manifests in multiple-choice benchmarks when answer options are shuffled (Vipul Gupta et al. in Changing Answer Order Can Decrease MMLU Accuracy), in automated fact-checking when claims and evidence are reordered (Payam Karisani in Fact Checking Beyond Training Set), and in pairwise semantic reasoning when the sequence of a premise and hypothesis is reversed (Weidi Xu et al. in Symmetric Regularization based BERT for Pair-wise Semantic Reasoning). Even when processing complex grammatical structures like comparative correlatives, models suffer from decision flips if the order of key terms is swapped (Leonie Weissweiler et al. in The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative).

The Thinking-Language Modeling Gap

Why do models exhibit such extreme sensitivity to order? The root cause lies in the tension between abstract logical reasoning and the sequential objectives of autoregressive language modeling.

Bo Han formalizes this in On the Thinking-Language Modeling Gap in Large Language Models, arguing that because models are optimized to predict the next word in human communication rather than the underlying logic of thought, they are inherently biased toward standard narrative progressions. When a premise appears after a conclusion in a sentence, or when the narrative flow deviates from a forward-chaining sequence, the model frequently overlooks the critical information.

This bias is reinforced by the causal attention mechanism of transformers. As Sangwon Yu et al. explain in Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context, causal attention naturally tracks evidence moving forward in time within the prompt. When supporting documents or clues are presented in an order that contradicts the logical chain of inference, the model's ability to synthesize information collapses because it cannot easily link disparate facts across non-sequential token spans.

This sequential dependency even dictates how models respond to prompting structures. Boshi Wang et al. (Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters) discovered that disrupting the coherence of steps in a Chain-of-Thought (CoT) demonstration by shuffling them prevents the model from effectively mimicking step-by-step reasoning. Similarly, forcing a model to output in structured schemas (like JSON) can impose a "reasoning tax" if the schema requires a conclusion before the explanation, effectively disrupting the natural linear flow of deduction (Zhi Rui Tam et al. in Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance).

Cognitive Bottlenecks: Entry Points and Attention Load

When premises are shuffled or presented out of order, the model faces several distinct processing hurdles that break its reasoning chain:

  1. Locating the Entry Point: In a structured forward-chaining problem, the starting point is clear. However, when facts are disordered, Shaotian Yan et al. (Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning) show that identifying the initial logical premise becomes a major bottleneck. Without a clear starting point, models often hallucinate intermediate steps to bridge perceived logical gaps between adjacent but unrelated sentences.
  2. Increased Attention Load: Randomly distributed premises force the model to look back and forth across its context window. Andrew Liu et al. (ReCogLab: a framework testing relational reasoning & cognitive hypotheses on LLMs) observe that this fragmentation increases the cognitive load on the attention mechanism, leading to frequent deduction errors. This suggests that even the internal geometric structures models use to represent relational hierarchies depend heavily on narrative flow rather than purely abstract relationships, as noted in Generalization from Starvation: Hints of Universality in LLM Knowledge Graph Learning.
  3. Greedy, Non-Backtracking Search: LLMs generate text token-by-token without an explicit global planning phase. Abulhair Saparov et al. (Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought) demonstrate that models act as "greedy reasoners." They excel at making individual forward steps when the context aligns perfectly with the proof path, but they struggle to search through context when information is presented in a non-sequential or bottom-up flow.

Heuristics and Human-Like Biases

Instead of executing robust deductive rules, models frequently rely on superficial shortcuts and positional heuristics embedded in the prompt structure.

For example, Vipul Gupta et al. found that models often exploit formatting patterns in evaluation datasets—such as numerical multiple-choice options listed in ascending order—as a shortcut to the correct answer. When these sequences are disrupted, accuracy plummets. Similarly, Nurul Fajrin Ariyani et al. (There's No Such Thing as Simple Reasoning for LLMs) show that "innocent" perturbations like duplicating premises or reordering facts cause a collapse in performance, confirming that the models are leveraging statistical shortcuts rather than stable logic. This reliance on word order over comparative logic is an architectural vulnerability that appears early in model development (Kyle Richardson et al. in Probing Natural Language Inference Models through Semantic Fragments), and standard language models fail to maintain consistency when rules are shuffled due to these positional heuristics (Alex Yuxuan Peng et al. in Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation).

Intriguingly, this behavior closely mirrors human cognitive limitations. Tiwalayo Eisape (A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models) notes that language models inherit structural biases from human language patterns, such as a sensitivity to the specific arrangement of terms within a syllogism (reminiscent of the psychological "figural effect"). Furthermore, Andrew Keenan Richardson et al. (Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning) show that reversing the order of premises can drastically shift the likelihood of a model committing specific logical fallacies, highlighting that the internal representation of a problem remains highly dependent on its surface textual structure.

Implications for Evaluation and Architecture

This order sensitivity exposes a profound gap between a model's ability to recognize individual facts and its ability to sequence them into a coherent reasoning chain (Dominic A. Simon et al. in Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysis). It also calls into question standard long-context benchmarks; as Hyeonseok Moon et al. demonstrate in NeedleChain: Measuring Intact Long-Context Reasoning Capability of Large Language Models, permuting logical sequences reveals that models struggle to integrate information out of chronological order even within relatively short context windows.

To move toward true, order-invariant reasoning, researchers are exploring dynamic evaluation frameworks like DyVal (Kaijie Zhu et al. in DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks), which tests models on reverse topological orders to bypass memorization. Ultimately, overcoming this brittleness requires moving beyond standard next-token objectives toward architectures or prompting frameworks that explicitly decouple logical dependencies from their linear narrative representations.

Key Foundational Work

Symmetric Regularization based BERT for Pair-wise Semantic Reasoning (2020) — contributed 2 concept elements to this direction.
Do Multi-hop Readers Dream of Reasoning Chains? (2019) — contributed 1 concept elements to this direction.
Probing Natural Language Inference Models through Semantic Fragments (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_premise_order_sensitivity_d52f7180,
  title = {premise order sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/premise-order-sensitivity/rdr_d52f7180a64642cf8656d59e658880a3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_premise_order_sensitivity_d52f7180_v1,
  title = {premise order sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/premise-order-sensitivity/rdr_d52f7180a64642cf8656d59e658880a3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake