Language models often look useless on a reasoning benchmark for most of pretraining and then improve abruptly. That jump makes it hard to forecast a large model's ability from small or partly trained ones. One useful fact helps: accuracy tracks pretraining loss far more smoothly than it tracks compute. Below a critical loss, accuracy rises roughly linearly; above it, accuracy sits at chance. This proposal asks whether the way we prompt a checkpoint moves that critical point.
The specific prompting method is an in-context curriculum, meaning a prompt whose worked examples step from trivially easy to target difficulty. Such curricula are known to rescue fully trained models: they let a 70B model hold a 10-back memory task, and they lift GSM8K accuracy sharply. Our hypothesis is that they also make capability visible earlier along the pretraining trajectory. If so, forecasts made from small, cheap checkpoints should become more accurate. The same experiments should also explain why published studies disagree on whether demonstration order matters.
Core Claim
Write $L$ for a checkpoint's pretraining loss on a fixed held-out corpus, $a_p(L)$ for its task accuracy under prompting strategy $p$, and $L^*_p$ for the loss below which $a_p$ departs from chance. We claim three things.
First, a well-built in-context curriculum raises $L^*_p$, so the departure from chance happens at a higher loss and therefore earlier in training. Second, the curriculum gain $\Delta(L) = a_{\text{curr}}(L) - a_{\text{static}}(L)$ depends on $L$ alone, collapsing onto one curve across model sizes. This mirrors the way normalized loss curves collapse across a model family. Third, because the curriculum curve leaves chance sooner, a loss-to-performance forecast fitted only on small or early checkpoints extrapolates more accurately to a larger held-out model when the evaluation uses curriculum prompts.
As a secondary claim, sensitivity to demonstration order is itself a function of $L$ and of task structure, not a fixed property of the prompting method. Order should matter when later steps reuse skills shown in earlier demonstrations. It should stop mattering when the difficulty tiers are independent.
Where This Sits
Our method comes from the trajectory side. Scaling Laws for Predicting Downstream Performance in LLMs maps FLOPs to loss and then loss to performance (the FLP approach), which beats fitting accuracy directly against compute. Understanding Emergent Abilities of Language Models from the Loss Perspective shows across 30 models that emergence happens at a critical loss, about 2.2 in its setup, even under continuous scoring. Both papers hold the prompt fixed. We treat the prompt as a variable and ask whether it moves the threshold.
Scaling Collapse Reveals Universal Dynamics and Scaling with Collapse supply the training recipe and the notion of collapse. Under matched tokens-per-parameter, schedule phase and the AdamW timescale, loss curves across model sizes fall onto one shape. We build our model ladder that way, so that loss is a clean clock shared across sizes.
The in-context curriculum literature motivates the work. Do Language Models Understand the Cognitive Tasks Given to Them? shows that n-back failures are task-comprehension failures that a 1-to-n curriculum fixes. Chain-of-Quizzes builds chains of examples the model can already solve, with complexity growing by length. Curriculum Demonstration Selection retrieves one demonstration per difficulty tier. Problem-Solving Logic Guided Curriculum ICL orders examples by counts of reasoning-step operations. All four test only finished models, and two of them contradict each other on ordering. Problem-Solving Logic finds that shuffling hurts; Curriculum Demonstration Selection finds it does not.
One paper is the direct constraint. Fine-Tune on the Format shows that intermediate checkpoints fail multiple-choice prompts because they lack symbol binding. A brief format fine-tune repairs this. That paper is both our hardest competitor and evidence that early-checkpoint signal is often hidden rather than absent.
Two nearer neighbours stop short. Adaptive Data Optimization tracks per-domain loss trajectories to shape the training-time data curriculum, not prompts. Human Curriculum Effects Emerge with In-Context Learning links in-context learning to weight learning only on small synthetic tasks, with no scaling analysis.
Research Program
We train a ladder of five decoder models, from roughly 50M to 1B parameters. All are trained under the scale-invariant controls of the Celerity recipe, with dense checkpoints throughout. This lets us confirm that normalized loss curves collapse before we rely on loss as the clock.
A sixth, larger model, a few billion parameters, is held out entirely as the forecasting target. Every checkpoint is evaluated under one fixed instrument: identical demonstrations, identical held-out loss corpus, and the same decoding settings. The primary curricula are fixed across checkpoints, so the instrument does not change while the model does. A checkpoint-adaptive variant, which reruns the Chain-of-Quizzes solvability filter at each checkpoint, is reported separately.
For each task and prompting strategy, we fit a piecewise accuracy-versus-loss curve. It is flat at chance above $L^*_p$ and linear below it, which is the form the emergence literature reports:
$$a_p(L) = c + \max(0,\; s_p\,(L^*_p - L)),$$
where $c$ is chance accuracy and $s_p$ is the slope after emergence. We estimate $L^*_p$ and $s_p$ with bootstrap intervals over checkpoints and seeds. We then test collapse of $\Delta(L)$ by comparing spread across model sizes at matched loss with spread across seeds within a size. This is the same noise-floor comparison used to define supercollapse.
The task suite spans four kinds of structure. GSM8K and MATH have natural difficulty tiers; MATH's tiers are what Curriculum Demonstration Selection uses. SciQ, SWAG and CopyColors isolate format and symbol binding. N-back gives a synthetic task where the curriculum is compositional: each level reuses the previous one. Parity-style Boolean tasks have controlled polynomial degree, following Generalization on the Unseen, Logic Reasoning and Degree Curriculum. That paper shows a minimum-degree bias, under which models default to low-degree shortcut solutions. It therefore gives a principled axis for when order should matter.
Baselines
The naive reference is static few-shot prompting with randomly drawn demonstrations, matched in count and token length to the curriculum prompt. This is what the FLP and emergence papers use. Chain-of-Quizzes reports 23.1% GSM8K accuracy for standard few-shot prompting against 88.6% for its curriculum, on a finished model.
The strongest existing forecasting method is FLP itself, run with static prompts on exactly our ladder and checkpoints. The FLP paper reports forecast error within 5% for a 7B model and 10% for a 13B model, using sampling models of at most 3B. That is the bar our forecast error on the held-out model must beat, at equal or smaller sampling-model budgets.
The strongest competitor for recovering early signal is format fine-tuning from Fine-Tune on the Format. On SciQ it reports 95.1% accuracy after auxiliary fine-tuning against 26.6% under a 3-shot baseline. We run it at every checkpoint with the paper's auxiliary data, and we report the extra compute alongside accuracy.
We also include similarity retrieval (KATE-style nearest neighbours), since Curriculum Demonstration Selection uses it as its comparison. Matching token budgets across all strategies keeps the comparison about structure rather than length.
The key ablation is the unordered curriculum: the same multi-difficulty demonstrations, shuffled. It isolates whether ordering matters at each loss level, or only the mix of difficulties. Beating the baselines means two things. Curriculum prompting must give a significantly higher $L^*$ than the static and shuffled prompts on at least the reasoning and n-back tasks. It must also give lower forecast error for the held-out model than FLP with static prompts.
Why not just…?
Why not just evaluate the final model with both prompts? For a model you already have, that is sufficient. It cannot tell a developer at 20% of training whether a capability is coming, and it cannot test whether the curriculum gain is a function of loss. Gaining that foresight is the point of the work. If $\Delta(L)$ fails to collapse across sizes, the forecasting claim fails, and we will report that.
Why not just fine-tune on the format? This is a real threat on SciQ and SWAG, where the bottleneck is plausibly symbol binding alone and fine-tuning may simply win. Our test is whether curriculum prompting approaches format fine-tuning there with zero gradient steps. The second question is whether it adds signal on GSM8K and n-back, where the deficit is task comprehension rather than answer format. If fine-tuning dominates everywhere, the proposal narrows to an ordering study.
Why not just use continuous metrics like Brier score? Understanding Emergent Abilities reports that the loss threshold persists even under Brier score and correct-choice probability. Smoother scoring therefore does not remove the jump. We will still report continuous metrics for every strategy, so readers can see whether the curriculum's effect survives them.
Why not just add more random demonstrations? Context-Scaling versus Task-Scaling in In-Context Learning proves that error falls as random demonstrations grow, but only asymptotically and for tasks the model already represents. Our length-matched random baseline, extended to twice the curriculum's token budget, tests directly whether quantity substitutes for structure at high loss.
Decisive Checks
The first check is threshold shift. On n-back and GSM8K, does curriculum $L^*$ exceed static and shuffled $L^*$ with non-overlapping bootstrap intervals?
The second is collapse. Is the cross-size spread of $\Delta(L)$ at matched loss within the seed noise floor?
The third is forecasting. Does the held-out model's accuracy, predicted from the three smallest ladder models, land closer under curriculum prompting than under FLP with static prompts?
The fourth is order sensitivity. Is the gap between ordered and shuffled curricula large on n-back and on high-degree parity, but near zero on MATH grade tiers? Does that gap shrink as $L$ falls? If so, the contradiction between Problem-Solving Logic and Curriculum Demonstration Selection resolves as a difference in task structure and model stage.
A clean negative is also informative. If curricula help only after the static threshold has already been crossed, in-context curricula amplify existing ability rather than reveal hidden ability, and early forecasting should not count on them.
References
- Scaling Laws for Predicting Downstream Performance in LLMs
- Understanding Emergent Abilities of Language Models from the Loss Perspective
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
- Scaling with Collapse: Efficient and Predictable Training of LLM Families
- Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm
- Chain-of-Quizzes: Pedagogy-inspired Example Selection in In-Context-Learning
- Curriculum Demonstration Selection for In-Context Learning
- Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning
- Fine-Tune on the Format: First Improving Multiple-Choice Evaluation for Intermediate LLM Checkpoints
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws
- Human Curriculum Effects Emerge with In-Context Learning in Neural Networks
- Generalization on the Unseen, Logic Reasoning and Degree Curriculum
- Context-Scaling versus Task-Scaling in In-Context Learning