When a language model is shown worked examples inside its prompt, the order of those examples sometimes matters a great deal and sometimes not at all. The published evidence disagrees. One study finds that shuffling an easy-to-hard sequence of reasoning demonstrations consistently lowers accuracy. Another finds that shuffling makes no difference, provided the examples span several difficulty levels. We propose a mechanistic account that could explain both results, and a way to test it with linear probes. A linear probe is a single linear classifier trained on a frozen model's internal activations to check whether some quantity can be read out of them.
The account rests on one structural fact. A decoder-only transformer can only look backward. An easy example placed before a hard one can therefore change how the model internally encodes the hard example. An easy example placed after it cannot. Our claim is that ordering helps only when the easy examples contain sub-steps of the hard ones. In that case the benefit is carried by a better encoding of the hard example's intermediate results, and that encoding can be measured with probes and transplanted between prompts. When the easy examples are merely easier, and not compositionally related, order should not matter, because there is nothing for them to scaffold. The claim predicts which of the two conflicting findings should appear in which setting, and it names where in the network the effect should live.
Core Claim
The disagreement is between Problem-Solving Logic Guided Curriculum In-Context Learning and Curriculum Demonstration Selection for In-Context Learning. The first retrieves demonstrations whose decomposed operations form sub-sequences of the target problem and orders them by step count. It reports that random ordering of the same examples lowers accuracy. The second picks one demonstration per metadata difficulty tier, such as grade level or LeetCode rating, and reports that shuffling does not hurt.
The two selection rules differ in one way that matters here. The first builds prompts in which easy examples are literal sub-computations of hard ones. The second builds prompts in which easy examples are only rated easier.
Our hypothesis is that curriculum order acts through the encoding of later demonstrations. The prediction is concrete. In compositionally nested prompts, a linear probe reading a hard demonstration's intermediate values should be more accurate when that demonstration comes after its sub-steps than when it comes before them. The accuracy gain should then be recoverable by patching those hard-demonstration activations into a shuffled prompt. In tier-diverse but non-nested prompts, neither the probe difference nor the behavioral order effect should appear.
This differs from existing work at the intersection. Prompt-Augmented Linear Probing uses a single static demonstration per class to make representations linearly separable for a trained classifier head. It never varies order or composition. Linear Probe Calibration fits an affine map on output probabilities only. How Do Transformers Learn In-Context Beyond Simple Functions? uses layerwise probes to separate early feature computation from later in-context regression, but with randomly sampled examples. None of these asks what ordering does inside the network, and none uses probes to decide an empirical dispute about curricula.
Research Program
We build prompts under a two-by-three design. The first factor is compositional relatedness. In the nested condition, easy demonstrations are sub-computations of the hard ones. In the matched condition, easy demonstrations are difficulty-matched to the nested ones but share no sub-operations with the hard ones. The second factor is order: easy-to-hard, shuffled, and hard-to-easy. Across all cells, the same demonstrations appear and total token length is held fixed.
We use three task families, chosen because their intermediate variables are known exactly and can serve as probe labels.
The first family is synthetic multi-step arithmetic and two-hop fact composition. Here the intermediate value, such as the result of step one, is generated alongside each problem.
The second is the n-back paradigm of Do Language Models Understand the Cognitive Tasks Given to Them?. Its in-context curriculum from 1-back to n-back is naturally nested. The intermediate variable is the identity of the item n positions back, and the matched control uses easier sequences from a different rule.
The third is GSM8K, where we use the operation decomposition of Ma et al. to define nesting and use annotated intermediate quantities as labels. MATH difficulty tiers, as used by CDS, supply the non-nested, tier-diverse condition in natural data.
For each prompt we record residual-stream activations at every layer. We take them at two places: the answer tokens of each hard demonstration, and the final token of the query. We train per-layer linear probes to decode the intermediate variable. Probes are always fit on activations pooled across all order conditions, so that differences reflect the representation and not a probe tuned to one condition. We report selectivity against a random-label control, following the linear-probe discipline that a flat hyperplane cannot manufacture structure the encoder lacks.
The causal test comes from Single-Position Intervention Fails, which constrains the design. That paper shows task identity can be decoded with 100% probe accuracy from single positions, yet single-position interventions produce 0% task transfer. Transfer reaches up to 96% only when all demonstration output positions are patched together at roughly 30% depth. We therefore never treat decodability as evidence by itself. Instead, we take all output-token activations of the hard demonstrations from an easy-to-hard run and patch them jointly, layer by layer, into the shuffled or hard-to-easy run of the same demonstration set. The demonstrations are identical, so each patched position corresponds to the same content. If accuracy on the query rises to the ordered level, the order effect is carried by how the hard demonstrations were encoded. If it does not, the effect lives elsewhere, for example in how the query attends to the whole context.
Models are Llama-3.2-3B, where the distributed-template result was obtained, and Llama 3.1 8B and 70B, where the n-back curriculum effect was reported.
Baselines
The naive reference point is the shuffled prompt built from the same demonstrations, together with a length-matched random few-shot prompt. Every order effect is measured against these, over many permutations and query samples per cell, so that the permutation sensitivity documented by Linear Probe Calibration is averaged out rather than mistaken for a curriculum effect.
The strongest existing method on the behavioral side is the logic-guided curriculum of Ma et al.. It reports a 2.24% average accuracy gain over the strongest prior in-context method on Llama3-8B across GSM8K and StrategyQA, using 2.74 examples on average. We reimplement its selection on GSM8K with the same demonstration pool, model, and example budget. Our nested condition should reproduce its ordered-over-shuffled gap. Failing to reproduce that gap would undercut the design before any probing.
The strongest tier-based competitor is Curriculum Demonstration Selection. It reports around 6% improvement on hard MATH problems and around 2% on easy ones, and states that shuffling does not hurt. We rerun it on MATH with its partitioning. Our prediction is that it will again show no order effect, and that its hard-demonstration probe readouts will not depend on order.
For n-back, the reference is the over 84% 10-back accuracy of Llama 3.1 70B with an in-context curriculum reported by Hu and Lewis. We know of no reported number for a shuffled or reversed version of that curriculum; producing one is part of this work.
The key ablation is the matched condition itself. It keeps difficulty spread, length, and example count identical to the nested condition and removes only the sub-operation overlap. A second ablation patches random, order-mismatched activations of the same norm, to rule out a generic perturbation effect.
Beating the account's rivals means two results together. First, a significant interaction: the ordered-minus-shuffled accuracy gap is larger in nested than in matched prompts. Second, the joint patch recovers a substantial fraction of that gap, while single-position patches and random-activation patches do not.
Why not just…?
Why not just run the ordered-versus-shuffled ablation behaviorally with more seeds and skip probing? This alternative is the most serious threat, and the behavioral half of our design does exactly this. The interaction between nesting and order can be established from accuracy alone, and it would already reconcile the two papers. What behavior cannot show is why the interaction occurs. Two mechanisms would produce the same accuracy pattern: better encoding of the hard demonstrations, or better query-time retrieval of the easy demonstrations. The patching experiment separates them, because only the first predicts that transplanting hard-demonstration activations recovers the gap. If both mechanisms turn out to contribute, we report that honestly rather than claiming the probe story.
Why not just use the model's own loss or perplexity on each demonstration instead of trained probes? Per-token loss on a hard demonstration is a reasonable cheaper proxy, and we report it. It measures how predictable the demonstration's text is, not whether its intermediate value is represented at the answer tokens. A demonstration can become more predictable through surface pattern-matching. Chain-of-Probe finds that about 20% of correct answers on ARC-Challenge rest on flawed intermediate reasoning, which is exactly the gap between output-level signals and internal computation that loss would miss. If demonstration loss tracks the order effect as well as the probes do, we will say so.
Why not just use a logit lens, which reads intermediate layers through the output vocabulary without training anything? A logit lens works when the intermediate variable is a single vocabulary token, and we include it as a readout baseline on the arithmetic tasks, where it applies. Many intermediate quantities, such as a partial sum inside a GSM8K solution or an item's position in n-back, are not expressed in output space at the middle layers. We also know that decodability without causal testing misleads, so neither the lens nor the probe is trusted without the joint patch.
Why not just make the probes non-linear to get higher decoding accuracy? Higher accuracy is not the goal. A multilayer probe can compute the intermediate value itself from raw activations, which would erase the contrast between ordered and shuffled encodings that we want to measure.
Proposed Experiments
The first decisive check is behavioral. On the synthetic tasks, n-back, and GSM8K, we test for the nesting-by-order interaction across the three models, with enough permutations per cell to estimate confidence intervals. The account fails outright if matched prompts show order effects as large as nested ones, or if nested prompts show none.
The second check is representational. At the hard demonstrations' answer tokens, the probe's accuracy for the intermediate variable should be higher after easy-to-hard ordering than after hard-to-easy ordering. The difference should appear only in the nested condition and should be concentrated in a band of layers. Given the distributed-template finding at roughly 30% depth, we expect that band in the lower-middle network, but we treat its location as an open measurement. At the query token, both orders may show similar decodability, since the query sees every demonstration regardless of order. Finding the difference at the demonstrations but not at the query would be the signature of the encoding account.
The third check is causal. Jointly patching the ordered hard-demonstration activations into the shuffled prompt should close most of the accuracy gap in nested prompts and do nothing in matched prompts. Single-position patches should fail, consistent with Single-Position Intervention Fails.
The last check reconciles the literature on its own benchmarks. Rerunning both selection methods under our shuffle protocol should reproduce the order sensitivity of Ma et al. and the order insensitivity of Curriculum Demonstration Selection, with the probe difference present only in the former. If that pattern holds, the practical advice follows directly: order demonstrations only when the easy ones are pieces of the hard ones, and otherwise choose them for coverage.
References
- Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning
- Curriculum Demonstration Selection for In-Context Learning
- Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-Shot In-Context Learners
- Enhancing In-context Learning via Linear Probe Calibration
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
- Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm
- Single-Position Intervention Fails: Distributed Output Templates Drive In-Context Learning
- Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Step