TL;DR
Humans and neural networks both suffer from a "simplicity bias," where they initially compress complex information into low-dimensional rules that ignore exceptions. This paper demonstrates that simple-to-complex curricula succeed by facilitating "curriculum unlearning"—helping the learner expand their representational manifold to accommodate the nuances they previously ignored.
What Problem Does This Paper Address?
While it is common knowledge in education and animal training that teaching simple rules before exceptions (a "start small" approach) is effective, we lack a robust computational explanation for why. In deep learning, the benefits of curriculum learning are notoriously inconsistent; often, a network trained on randomly shuffled data performs just as well as one trained with a carefully sequenced curriculum.
The authors investigate the gap between biological and artificial learning by asking: Does the benefit of a curriculum lie in how it shapes the internal geometry of a learner's representations? They propose that curricula are not just about "learning" new things, but about "unlearning" the overly simple, low-dimensional solutions that learners naturally gravitate toward early in training.
Methods
The researchers used a yoked experimental design, comparing human behavior directly with neural network models:
- The Task: A "Rule plus Exception" (R+E) classification task. Participants (and networks) categorized "bug" stimuli based on three binary features (e.g., wing shape, body pattern). Six stimuli followed a simple 1-D rule, while two were exceptions.
- Human Study: 207 participants in Experiment 1 and 1,027 in Experiment 2.
- Neural Network Modeling: Two-layer feedforward networks (32 ReLU hidden units) were trained using "yoked" sequences—each network received the exact same trial-by-trial sequence and feedback as a specific human counterpart.
- Curriculum Conditions:
- No Curriculum: Randomly interleaved rule and exception stimuli.
- Simple-to-Complex: Rule stimuli presented early, with exceptions gradually introduced.
- Complex-to-Simple: Exceptions presented early, then phased out.
- Representational Analysis: The authors measured the "Participation Ratio" (a proxy for dimensionality) in networks and used Reaction Time (RT) costs in humans as a behavioral proxy for representational compression.
Datasets
The study utilized synthetic datasets of "bug" stimuli defined by three binary features (8 unique stimuli total).
- Experiment 1: Focused on the basic R+E task under different curricula.
- Experiment 2: Introduced a pretraining phase where learners were exposed to either a 1-D rule (inducing high compression) or an XOR rule (inducing low compression) before tackling the main R+E task.
Key Insights and Ideas
- Simplicity Bias is Universal: Both humans and networks show a path-dependent learning trajectory, mastering 1-D rules long before exceptions. In networks, this corresponds to a literal compression of the hidden layer representations onto a single axis.
- Curriculum as "Uncompression": The simple-to-complex curriculum helps learners escape the "trap" of a low-dimensional solution. By gradually introducing exceptions, the curriculum encourages the representational manifold to expand (uncompress) along new, task-relevant dimensions.
- The Cost of Starting Hard: The "Complex-to-Simple" curriculum actually harmed learning. Learners often "overwrote" the exceptions they saw early once the dominant rule became apparent, leading to catastrophic forgetting.
- Priors Matter: The benefit of a curriculum is greatest when the learner starts with a highly compressed prior. Experiment 2 showed that participants pretrained on simple 1-D rules benefited significantly more from a curriculum than those pretrained on complex XOR rules.
Key Results
- Performance Boost: In humans, the simple-to-complex curriculum significantly improved test accuracy on exceptions compared to no curriculum (Log-Odds = 0.78, p = .008).
- Dimensionality: Networks trained with a simple-to-complex curriculum reached a higher final representational dimensionality (Participation Ratio = 2.37) compared to those with no curriculum (2.27).
- Human RT Signature: In humans, the "cost" of processing exceptions (slower reaction times) only emerged later in training, mirroring the "uncompression" phase seen in the neural networks.
- Pretraining Interaction: In Experiment 2, the curriculum benefit for exception stimuli was nearly three times larger for networks (and significantly larger for humans) that arrived at the task with a low-dimensional prior induced by 1-D pretraining.
Limitations and Future Work
- Architectural Simplicity: The neural network used was a basic MLP. The authors note that biological systems have a distinct division between the hippocampus (good for exceptions) and the neocortex (good for rules), which this model lacks.
- Forgetting: The networks showed a slight degradation in rule performance when learning exceptions that was not observed in humans, suggesting human learning is more robust to interference.
- Future Directions: The authors suggest that future models incorporating hippocampal-like structures might provide an even more faithful account of how curricula facilitate the "unlearning" of simplicity biases.
Summary updated