Is the Curriculum in the Data or in the Network? Linear-Signal Bootstrapping for Learning Non-Linear Tasks

Transformers trained to learn quadratic functions "in context" (inferring a new function from example input–output pairs in the prompt) fail completely when trained on quadratics alone. The same models succeed in a majority of runs when linear functions are mixed in first. A natural reading is that networks need to start "linear" before they can become non-linear. This reading appears in two forms. One changes the data: easy-to-hard task curricula. The other changes the architecture: smoothly blending a non-linear activation with the identity, as in activation annealing. Both forms rest on the same intuition, which the theory of deep linear networks seems to support: linear models have well-behaved loss surfaces, and non-linear models do not.

This note proposes that the intuition is aimed at the wrong object. The hypothesis is that what a curriculum supplies is not a linear network but a linear signal. That signal is a component of the training targets that a linear in-context rule can already reduce. Gradient descent uses it to build the attention circuit that later carries the non-linear task. If this is right, the success of a curriculum is predicted by a single measurable quantity: the fraction of target variance that a linear predictor can explain, tracked over training. Making the network itself linear early should then be useless whenever that fraction is near zero. The proposal tests this directly by pitting a data curriculum, an activation curriculum, and a "signal-leakage" curriculum against each other. The signal-leakage curriculum keeps only quadratic tasks but temporarily adds a small linear term to them.

Core Claim

The motivating evidence comes from How does Multi-Task Training Affect Transformer In-Context Capabilities?. That paper trains a 22.4M-parameter GPT-2 on in-context regression over Hermite-polynomial function classes: linear $F_1$, quadratic $F_2$ and cubic $F_3$. It reports a 0% convergence rate for single-task $F_2$ training. A mixed curriculum ($F_1$, then $F_1{+}F_2$, then all three) lets 60% of models master $F_2$ using 1/9th of the training data. The same "retrospective" attention heads, which map each prompt input to its paired output, serve all three classes. The paper offers no account of why the linear phase helps.

Our account rests on one property of Hermite polynomials. Under Gaussian inputs, the degree-two Hermite polynomial $x^2-1$ is uncorrelated with $x$. A pure-quadratic target therefore gives an in-context linear rule zero expected loss reduction. A gradient that would start building the retrospective heads gets essentially no push from such a target.

To make this measurable, define the linear signal fraction at training step $t$ as

$$\rho(t) = \frac{\mathbb{E}_{f \sim P_t}\,\mathrm{Var}[\,\ell_f(x)\,]}{\mathbb{E}_{f \sim P_t}\,\mathrm{Var}[\,f(x)\,]},$$

where $P_t$ is the distribution of training functions at step $t$, and $\ell_f$ is the best linear approximation of function $f$ under the input distribution. In words, $\rho(t)$ is the share of target variance a linear predictor can capture.

The core claim has two parts. First, whether a run converges on $F_2$ depends mostly on the early-training trajectory of $\rho(t)$, and much less on how that trajectory is produced. Second, a linear-to-non-linear activation schedule changes the network but leaves $\rho(t)$ unchanged. It should therefore not reproduce the curriculum benefit on pure quadratics, despite what deep-linear-network intuition suggests.

If the paper's $F_2$ class contains degree-one terms, single-task failure cannot be due to a missing linear signal. It would instead be due to a low $\rho$: the linear component is drowned out by quadratic variance. The prediction then becomes a threshold in $\rho$ rather than its mere presence. We will build both "pure" and "full" variants of $F_2$ so the experiment covers either case.

Relation to Prior Work

The closest architectural work is AFA-LoRA. It interpolates an adapter activation as $\beta(t)\sigma(x) + (1-\beta(t))x$ and anneals $\beta$ from 1 to 0 so that low-rank adapters stay mergeable at inference. It reports 86.34% average accuracy for AFA-DoRA on Commonsense-170K with Llama-3-8B, against 87.07% for full fine-tuning. It never runs the reverse schedule, from linear to non-linear, as an optimization curriculum. We supply that reverse schedule as a competing hypothesis, not as our method. Our claim is that it fails exactly where $\rho \approx 0$.

The theory that makes the reverse schedule tempting comes from deep linear networks. Bartlett et al. prove that gradient descent from identity initialization efficiently learns positive definite maps. Chou et al. show singular values being recovered one after another in a "waterfall." Yun et al. characterize implicit biases toward minimum-norm and max-margin solutions. These results constrain our proposal more than they support it. Bartlett et al. also prove that from identity initialization, targets with a negative eigenvalue are never reached, with loss bounded below by $\lambda^2/2$. Linearity therefore guarantees good behavior only for targets that linear dynamics can already express. That is the same "the starting regime must already carry signal" logic we apply to targets.

On the curriculum side, the task-centric theory of easy-to-hard self-improvement proves that easy-to-hard curricula beat fixed mixtures through a data-efficiency feedback loop. It also finds that a model starting outside a feasible region regresses: −0.28% for Llama-3.2-1B on GSM8K. Our $\rho$ is a candidate concrete measure of that feasible region for gradient-based in-context learning.

For tools, we take the two-stage picture from From Condensation to Rank Collapse. In that picture, key-query matrices stay dormant during an early loss plateau and then collapse to low rank. We use it to define when the in-context circuit forms.

For a second testbed we use On Mesa-Optimization in Autoregressively Trained Transformers. It proves that linear attention trained on $x_{t+1}=Wx_t$ implements one internal step of least-squares regression, recovering about 1/5 of the true signal on dense Gaussian data and all of it on sparse data.

Research Program

We first reproduce the Hermite setup at the original model size. We record per-seed convergence on $F_1$, $F_2$ and $F_3$ under three conditions from the original paper: single-task training, random mixture, and mixed curriculum. We also compute $\rho(t)$ exactly from the sampling distribution; this is cheap because Hermite components are orthogonal.

We then add two interventions that separate "linear network" from "linear signal."

The activation curriculum trains on pure $F_2$ only. Every MLP non-linearity, and optionally the softmax, which we blend with a linear normalization, starts at $\beta=0$ and anneals to $\beta=1$. We test several schedule lengths.

The leakage curriculum also trains only on quadratic tasks. Each target is replaced by $f(x) + \epsilon(t)\, a^\top x$, with a random per-prompt vector $a$ and $\epsilon(t)$ annealed to zero. We tune the schedule so that $\rho(t)$ tracks the trajectory produced by the mixed curriculum. If convergence is governed by $\rho(t)$, leakage should match the mixed curriculum's success rate. It would do so without any separate linear task and without any architectural change. The activation curriculum should match single-task failure.

To explain why outcomes differ, we log three quantities throughout training. The first is the step at which retrospective heads emerge, identified with the original paper's attention-pattern criterion. The second is the effective rank of the key-query product, which marks the transition from condensation to rank collapse. The third is the fraction of GeLU pre-activations in the activation's nearly linear range, which measures how linear the network actually is.

The mechanism predicts that head emergence occurs during the high-$\rho$ phase and precedes the drop in $F_2$ loss. It also predicts that, at matched $\rho(t)$, the network's own degree of linearity does not predict success.

Finally, we fit a simple model of convergence probability as a function of early-phase average $\rho$, using sweeps of constant-$\rho$ training. We then check whether this fit predicts the success of held-out schedules.

Baselines

The do-nothing reference is single-task $F_2$ training with the original hyperparameters. The reported number is a 0% convergence rate (Bhasin et al.). Any intervention must beat this on matched data budgets.

The strongest existing method is the mixed curriculum from the same paper, with 60% of models mastering $F_2$ at 1/9th of the data. We will rerun it ourselves with the same seeds, token budget and tuning budget as every other arm, because our success threshold must be applied identically. The leakage curriculum "beats" it scientifically if it reaches comparable success, within seed-level confidence intervals, without linear tasks. It beats it practically if a $\rho$-designed schedule reaches the same success with fewer tokens.

The random uniform mixture of $F_1$, $F_2$ and $F_3$ from step zero is the standard large-scale practice. It isolates ordering from mere inclusion of linear tasks. Under our hypothesis it should perform roughly in line with its constant $\rho$.

The reversed activation homotopy, from linear to non-linear, is the architectural competitor suggested by the open gap in AFA-LoRA. For it, no reported result was found on Lacuna. It gets the same schedule-length search as the leakage arm. It wins if it converges on pure $F_2$ at rates near the mixed curriculum.

Two optimizer controls follow. Learning-rate warmup with small initialization, tuned over the same grid, tests whether the network's natural early near-linearity already does the job. Sharpness-aware Minimization is run at matched wall-clock time, since it doubles per-step cost. Its reported gains, such as ImageNet ResNet-152 top-1 error dropping from 20.3% to 18.4%, concern generalization rather than whether training converges at all.

The key ablation is leakage with $\rho(t)$ matched to the mixed curriculum versus leakage with $\rho(t)$ deliberately mismatched, starting late or staying too low. That ablation isolates $\rho$ as the active ingredient.

Why not just…?

Why not just make the network linear early, as deep-linear-network theory suggests? That is the reversed AFA schedule. We run it as a full arm rather than dismiss it. The Hermite orthogonality argument predicts that a linear model gets no expected gradient toward the in-context circuit from pure-quadratic targets. Bartlett et al.'s negative-eigenvalue barrier is an analogous case where linear dynamics stall on targets they cannot express. If the activation arm succeeds on pure $F_2$, our hypothesis is wrong, and we would report the architectural curriculum as the finding.

Why not just use warmup and small initialization, which already make GeLU networks nearly linear early? This is a real threat to any architectural story, which is why we log the near-linear fraction of pre-activations. Like the activation curriculum, it leaves $\rho$ unchanged. We therefore predict it fails on pure $F_2$. The original paper's tuned single-task runs reaching 0% are consistent with this, but they do not settle it, so we tune these controls on the same grid as everything else.

Why not just use a flatter-minimum optimizer like SAM? SAM reshapes the step toward flat regions. It cannot create a gradient component that the targets do not provide. The 0% single-task result is a failure to start, not a failure to generalize. A matched-compute SAM arm on pure $F_2$ tests this directly.

Why not just keep using the mixed curriculum, since it works? That curriculum depends on having a ready-made family of easier tasks. The point of $\rho$ is to design a schedule from properties of the target distribution alone, and to predict when a curriculum is unnecessary. If leakage does not match the mixed curriculum, this justification fails. We would then conclude that easier tasks carry something beyond linear signal, such as distinct prompt statistics.

Decisive Checks

The first decisive check is the three-way comparison on pure $F_2$ across at least 20 seeds per arm, with success defined by normalized mean-squared-error thresholds as in the original paper. The hypothesis survives only if leakage clearly exceeds single-task and activation-curriculum success, and also approaches the mixed curriculum.

The second check is whether the fitted relation between convergence and early-phase $\rho$ correctly ranks held-out schedules. These include cubic tasks, where the hypothesis says the curriculum should supply linear signal for $F_3$ through its degree-one Hermite projection.

The third check transfers the logic to the autoregressive testbed of Zheng et al., with dimension 5 and context length 100. We train a softmax Transformer on dense Gaussian sequences, preceded either by a sparse-to-dense data schedule or by a linear-to-softmax attention homotopy. Recovery is measured as the fraction of the true transition matrix captured, against the reported baseline of about 20% on dense Gaussian data versus 100% on sparse data. Here sparse sequences play the role of a high-signal phase. The hypothesis predicts that the data schedule narrows the gap while the attention homotopy does not.

Agreement across both testbeds would support the claim that early-phase linear signal in the targets, rather than early linearity of the network, is what the curriculum supplies. A failure in either testbed would bound the claim to the regime where it holds.