When Prerequisites Gate Learning: A Solvable Large-System Theory of the Curriculum Threshold

Neural networks often cannot learn a composite skill unless they are also shown the simpler skills it is built from. The clearest demonstration uses a synthetic benchmark of bit-string puzzles. When networks were trained only on the composite puzzles, 0 of 40 converged. When the atomic sub-puzzles were added to the training mix, most converged quickly (On the creation of narrow AI). Nobody can yet say how much atomic data is enough, how that amount grows with problem size, or how failure spreads through a deep tree of prerequisites. The only exactly solvable theory of skill emergence, An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem, gets its tractability by assuming that skills never interact. In that model a composite skill is just a rarer skill, and it eventually gets learned no matter what else is in the data.

We propose a solvable model in which skills are coupled through a logical AND. A composite skill's output is the product of its prerequisites' outputs, so the composite can only receive a useful learning signal once its parents are partly learned. We analyze this model in the large-system limit, borrowing the tools wireless-communications theory uses for systems with many interacting users. The model makes two concrete predictions. First, there is a sharp critical fraction of atomic data, $\rho^*$, below which composite skills stay unlearned for the whole training budget, and its scaling with input size and parity order can be derived. Second, in a large random prerequisite graph, mastery spreads as a deterministic front moving from shallow to deep skills, and its speed can be computed from a low-dimensional recursion. Both predictions are checked against trained MLPs and Transformers.

Core Claim

Treating a prerequisite as a multiplicative gate, rather than as a frequency difference, changes the qualitative behavior of learning. In the decoupled theory, a composite skill's loss falls eventually under any data mix, at a rate set by how often that skill appears. In the gated theory, gradient-based learning near a small initialization has a plateau whose length grows polynomially with input dimension. Atomic data shortens this plateau, and the transition from "never learned within budget" to "learned" is sharp in the large-system limit. This yields a formula for the critical atomic fraction that can be calibrated on small runs and tested on larger ones. That is the step from the empirical 0-of-40 observation to a predictive law.

Two kinds of prior evidence motivate the multiplicative gate. In tutoring data, a learned composition model with no structural constraint converged on its own to an AND gate: knowing an advanced skill without its prerequisite gave an effective mastery of about 0.05, against about 1.0 with both (The Composition Effect). The narrow-AI paper shows the same gating in networks, though only empirically. The method comes from large-system analysis. The decoupling principle of Microscopic analysis for decoupling principle of linear vector channel shows that in a large coupled system, any small group of components behaves as if it faced independent scalar channels whose parameters are set by system-wide averages. We reuse this idea so that each skill sees an effective drive determined by the average mastery of its parents. One caution constrains the analysis. The cited decoupling result is for linear systems with additive noise, and an AND gate is neither linear nor additive. We therefore do not assume Gaussian interference. We derive the effective drive directly from the product structure, and we verify concentration numerically rather than claiming it.

The Model

Each atomic skill $i$ is a learned module with mastery amplitude $m_i \in [0,1]$. Each composite skill $c$ reads the product of its parent modules, scaled by its own amplitude $w_c$. Let $\rho$ be the fraction of training samples that come from atomic tasks. The schematic gradient-flow dynamics in the reduced coordinates are

$$\dot m_i = \Big[\rho\,\pi_i + (1-\rho)\sum_{c \ni i} \pi_c\, w_c \prod_{j \in P(c),\, j\neq i} m_j \Big]\, G_n(m_i),$$

where $\pi_i$ and $\pi_c$ are the task frequencies within each pool and $P(c)$ is the set of parents of $c$. The term $G_n(m)$ is the rate at which a module that currently has mastery $m$ is improved, and it is small near $m=0$ when the input dimension $n$ is large. This is where the difficulty of locating a sparse parity's support enters. Without the product term the model reduces to the decoupled multilinear model of Biroli et al., which gives a clean ablation. Including the product term creates a degenerate saddle point at small amplitudes. Atomic data adds a term linear in $\rho$ that breaks this degeneracy.

The first analytical target is the plateau length $T(\rho, n, k)$ for composites built from order-$k$ parities, together with the boundary $\rho^*(B)$ at which $T$ equals the training budget $B$. Our working hypothesis is $\rho^* B \propto n^{\,k-1}$. This hypothesis comes from the commonly assumed cost of finding a $k$-sparse parity by gradient descent, divided by the share of atomic data. Deriving the exponent inside the model, instead of assuming it, is part of the work. Measuring how the transition sharpens as $n$ grows is the large-system statement.

The second target is a deep hierarchy. We take $S$ skills arranged in a random directed acyclic graph with a fixed distribution of the number of parents, and let $S \to \infty$. If the graph is locally tree-like, the decoupling argument implies that the mastery $q_\ell(t)$ at depth $\ell$ obeys a deterministic recursion. Each level's drive depends on the parents' $q_{\ell-1}(t)$ raised to the in-degree, averaged over the in-degree distribution. This recursion predicts a mastery front whose speed depends on depth, in-degree, and $\rho$. It also predicts how Zipf-distributed task frequencies interact with depth. Here the theory should depart from the $-\alpha/(\alpha+1)$ time and data exponents of the decoupled model.

Research Program

The work has three parts. The first is analytical: solve the two-level gated dynamics, obtain $T(\rho,n,k)$ and $\rho^*$, and derive the depth recursion and its front speed. The second is a set of synthetic generators. We extend Compositional Multitask Sparse Parity (CMSP) so that $\rho$, $n$, $k$, prerequisite depth, in-degree, and task-frequency exponent can each be set independently. The third is a calibrate-then-predict protocol in the style of Biroli et al. We fit the model's few constants on the smallest configuration only, then predict, before training, the convergence boundary and front speed for larger $n$, deeper graphs, and different architectures.

Baselines

The naive reference is uniform mixing with no deliberate atomic share. Its known outcome on isolated composite training is 0 of 40 converged networks (On the creation of narrow AI). We rerun it with 40 seeds per configuration so it uses the same budget as every other condition.

The strongest existing predictive theory is the decoupled multilinear model of An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem. It predicts sequential emergence by frequency, with loss exponents of $-\alpha/(\alpha+1)$ in time and data and $-\alpha$ in parameters. It is the right competitor because it is the only theory in the file that already predicts emergence times quantitatively from calibration on one skill. We give it the same calibration data and the same number of free constants, treating each composite as an independent rarer skill. We beat it if its predicted convergence times for composites are off by more than a factor of two in the low-$\rho$ regime while ours stay within that factor, or if it predicts eventual convergence where all seeds fail.

The strongest existing curriculum method is the online mirror-descent scheduler of Skill-it!. It reorders data mixtures using a skills graph measured from pilot runs, and it reported a 36.5-point accuracy gain over random sampling on LEGO. There is no reported result on Lacuna for this scheduler on CMSP. We run it on CMSP with its pilot-run cost counted against the same total compute. We compare its final composite accuracy and steps to convergence with a static mixture set at the theory's predicted $\rho^*$ plus a safety margin. The theory-derived mixture uses no pilot runs. Matching the scheduler within seed noise counts as a win. Losing to it by a wide margin means the theory captures the threshold but not the optimal schedule.

The key ablation removes the product term, keeping the gated model's architecture and calibration but setting the composite drive to be independent of parent mastery. This isolates the AND coupling as the source of any predictive gain.

Why not just…?

Why not just sweep $\rho$ empirically and read off the threshold? For one configuration, a sweep is cheaper and fully adequate. This is a real threat to the practical value of the work. The theory earns its place only through extrapolation: predicting the boundary at larger $n$, larger $k$, and deeper graphs from the smallest configuration. The calibrate-then-predict experiment settles this directly. If the theory's extrapolations are no better than fitting a power law to small sweeps, we will report that the gating model adds explanation but not prediction.

Why not just treat composites as rare skills in the existing solvable model? That model has no mechanism for a composite to stay unlearned indefinitely. Its plateaus shrink as the skill's frequency rises, regardless of what else is present. The decisive check is the regime with no atomic data and composite frequency pushed high. The decoupled model predicts fast learning there. The gated model predicts a plateau that grows with $n$, which is consistent with the reported 0 of 40.

Why not just use the measured skills graph and a scheduler, with no theory at all? Skill-it! shows schedulers work, but its own stated cost is pilot training to measure the graph's edges. It also offers no account of when a mixture will fail outright. Our comparison charges the scheduler for its pilot compute. If it still wins clearly, the proposal shrinks to an explanatory model of its success.

Proposed Experiments

The first experiment is a two-level phase diagram on extended CMSP. We use 2-layer MLPs and small Transformers, with $n$ from roughly 30 to 120, $k \in \{2,3,4\}$, and $\rho$ on a log grid. For each cell we record the fraction of 40 seeds that converge within budget $B$. We overlay the predicted boundary $\rho^*(B,n,k)$ and test whether the measured transition width shrinks with $n$, as a large-system limit should.

The second experiment uses deep random prerequisite graphs with hundreds of skills. We compare measured per-depth mastery curves with the deterministic recursion and check concentration across seeds as $S$ grows. Under Zipf frequencies, we test whether the loss exponent shifts with depth relative to $-\alpha/(\alpha+1)$.

The third experiment is the curriculum comparison described in the baselines section.

The single most decisive check is the no-atomic, high-frequency composite regime. If composite loss falls on the decoupled model's predicted schedule there, the gating hypothesis is wrong and the project changes direction. If loss plateaus for a time that scales with $n$ as predicted, the threshold theory stands.