Curriculum unlearning in humans and neural networks

Jirko Rubruck, Alice Zhang, Andrew Saxe, Christopher SummerfieldarXiv 2026 · arXiv

TL;DR

Humans and neural networks both suffer from a "simplicity bias," where they initially compress complex information into low-dimensional rules that ignore exceptions. This paper demonstrates that simple-to-complex curricula succeed by facilitating "curriculum unlearning"—helping the learner expand their representational manifold to accommodate the nuances they previously ignored.

What Problem Does This Paper Address?

While it is common knowledge in education and animal training that teaching simple rules before exceptions (a "start small" approach) is effective, we lack a robust computational explanation for why. In deep learning, the benefits of curriculum learning are notoriously inconsistent; often, a network trained on randomly shuffled data performs just as well as one trained with a carefully sequenced curriculum.

The authors investigate the gap between biological and artificial learning by asking: Does the benefit of a curriculum lie in how it shapes the internal geometry of a learner's representations? They propose that curricula are not just about "learning" new things, but about "unlearning" the overly simple, low-dimensional solutions that learners naturally gravitate toward early in training.

Methods

The researchers used a yoked experimental design, comparing human behavior directly with neural network models:

  • The Task: A "Rule plus Exception" (R+E) classification task. Participants (and networks) categorized "bug" stimuli based on three binary features (e.g., wing shape, body pattern). Six stimuli followed a simple 1-D rule, while two were exceptions.
  • Human Study: 207 participants in Experiment 1 and 1,027 in Experiment 2.
  • Neural Network Modeling: Two-layer feedforward networks (32 ReLU hidden units) were trained using "yoked" sequences—each network received the exact same trial-by-trial sequence and feedback as a specific human counterpart.
  • Curriculum Conditions:
    1. No Curriculum: Randomly interleaved rule and exception stimuli.
  • Simple-to-Complex: Rule stimuli presented early, with exceptions gradually introduced.
  • Complex-to-Simple: Exceptions presented early, then phased out.
  • Representational Analysis: The authors measured the "Participation Ratio" (a proxy for dimensionality) in networks and used Reaction Time (RT) costs in humans as a behavioral proxy for representational compression.
Figure 1: a, Participants learned to classify bug stimuli as either “good” or “bad” in a rule plus exception (R+E) task. Most bugs can be correctly classified based on a single feature (here wings) but there are two exceptions to this rule. b, Participants learned the task in a trial-by-trial manner with fully supervised feedback. After the training phase, they were assessed in a test phase without feedback. c, We modelled the task using a two-layer neural network with stimuli (d) which similarly varied along three binary dimensions. The eight stimuli can be depicted as the corners of a cube in a three-dimensional binary feature space. Colour denotes rule (purple) versus exception (blue) stimuli and shape denotes the category label (circle = Class 0, triangle = Class 1). The network (bottom) contained H ReLU hidden units and a sigmoid output unit and was trained using trial-wise gradient descent, we trained one network per participant on that participant's own sequence of trials, so that each network receives identical stimuli in identical order to human participants e, Training consisted of five learning blocks of 48 trials each in which rule and exception stimuli were randomly interleaved. In the “no curriculum” condition, stimuli were sampled pseudorandomly on each block (36 rule stimuli and 12 exceptions). In the simple-to-complex condition, the fraction of exception stimuli in each block increased over the course of five blocks from 0 to 0.5. In the “complex-to-simple” curriculum, the fraction of exception stimuli decreased over learning blocks. All curriculum conditions are matched for the overall number of exposures to each stimulus.

Datasets

The study utilized synthetic datasets of "bug" stimuli defined by three binary features (8 unique stimuli total).

  • Experiment 1: Focused on the basic R+E task under different curricula.
  • Experiment 2: Introduced a pretraining phase where learners were exposed to either a 1-D rule (inducing high compression) or an XOR rule (inducing low compression) before tackling the main R+E task.

Key Insights and Ideas

  • Simplicity Bias is Universal: Both humans and networks show a path-dependent learning trajectory, mastering 1-D rules long before exceptions. In networks, this corresponds to a literal compression of the hidden layer representations onto a single axis.
  • Curriculum as "Uncompression": The simple-to-complex curriculum helps learners escape the "trap" of a low-dimensional solution. By gradually introducing exceptions, the curriculum encourages the representational manifold to expand (uncompress) along new, task-relevant dimensions.
  • The Cost of Starting Hard: The "Complex-to-Simple" curriculum actually harmed learning. Learners often "overwrote" the exceptions they saw early once the dominant rule became apparent, leading to catastrophic forgetting.
  • Priors Matter: The benefit of a curriculum is greatest when the learner starts with a highly compressed prior. Experiment 2 showed that participants pretrained on simple 1-D rules benefited significantly more from a curriculum than those pretrained on complex XOR rules.
Figure 2: a, Proportion of participants (n = 207) classified into each of five strategies at the end of each learning block and at test, separately by curriculum. Strategies were assigned by fitting nine deterministic policies plus a lapse rate and selecting by BIC against an unpenalised random-responding null. b, The same classification applied to networks, one per participant, trained on that participant's exact trial sequence. Icons mark which rows show people and which show networks. c, Block-to-block flow of human strategies, with ribbon width proportional to the number of participants making each transition; the dotted vertical line marks the test phase. d, Transition matrices summarising the transitions in c, across training blocks giving the percentage of participants in each state at block b who occupied each state at block b+1. Rows sum to 100%. Pooled across curricula, the odds of moving from rule to rule + exceptions were 2.23 times higher under the simple-to-complex curriculum than under the other two (95% CI [1.18, 4.24], p = .014; Mantel–Haenszel stratified by block, participant-clustered bootstrap [1.21, 4.51]).

Key Results

  • Performance Boost: In humans, the simple-to-complex curriculum significantly improved test accuracy on exceptions compared to no curriculum (Log-Odds = 0.78, p = .008).
  • Dimensionality: Networks trained with a simple-to-complex curriculum reached a higher final representational dimensionality (Participation Ratio = 2.37) compared to those with no curriculum (2.27).
  • Human RT Signature: In humans, the "cost" of processing exceptions (slower reaction times) only emerged later in training, mirroring the "uncompression" phase seen in the neural networks.
  • Pretraining Interaction: In Experiment 2, the curriculum benefit for exception stimuli was nearly three times larger for networks (and significantly larger for humans) that arrived at the task with a low-dimensional prior induced by 1-D pretraining.
Figure 3: Simple-to-complex curricula boost exception learning in humans and neural networks. a, Human accuracy over training, binned into ten 24-trial bins, separately for rule (purple) and exception (blue) stimuli and for each curriculum. Shading gives ±1 SEM across participants. b, Human test accuracy, averaged over two presentations of each of the eight stimuli. c, d The same for the networks, measured identically, per-trial accuracy on the presented stimulus during training, and noise-marginalised accuracy at test. Networks inherit each participant's trial sequence, so the two rows are matched trial for trial and differ only in the learner. Error bars give ±1 SEM. e, f Hyperparameter sweep for the networks. Colour gives the curriculum benefit — simple-to-complex minus no curriculum — in end-of-training accuracy on exception stimuli (e) and rule stimuli (f). The black box marks the configuration used throughout the paper. The exception benefit is positive in 34 of 35 configurations, so the effect does not depend on the exact hyperparameters selected.

Limitations and Future Work

  • Architectural Simplicity: The neural network used was a basic MLP. The authors note that biological systems have a distinct division between the hippocampus (good for exceptions) and the neocortex (good for rules), which this model lacks.
  • Forgetting: The networks showed a slight degradation in rule performance when learning exceptions that was not observed in humans, suggesting human learning is more robust to interference.
  • Future Directions: The authors suggest that future models incorporating hippocampal-like structures might provide an even more faithful account of how curricula facilitate the "unlearning" of simplicity biases.

Summary updated

Abstract

During learning, humans often benefit from curricula that teach simple rules before exceptions. However, we lack a computational explanation for this phenomenon. Here, using a canonical behavioural task and modelling with neural networks, we develop and test a theory for why simple-to-complex curricula succeed. We show that during learning, both humans and neural networks show simplicity biases, a tendency to learn simple rules that are underfit to the data. In neural networks this bias is accompanied by compression of neural representations. Simple-to-complex curricula help both types of learners overcome this compression, developing representations (and policy) with appropriate dimensionality for the task. We test and confirm a further prediction from this theory: learners that have been pre-exposed to simple tasks and with more compressed initial representations benefit more from curricula. These findings suggest that curricula help with ``unlearning'' of simple rules.
BibTeX
@misc{rubruck2026curriculum,
  author = {Jirko Rubruck and Alice Zhang and Andrew Saxe and Christopher Summerfield},
  title = {{Curriculum unlearning in humans and neural networks}},
  year = {2026},
  eprint = {2611.04565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04565},
}
Figure