Abstract
Researchers and educators have long wrestled with the question of how best to teach their clients be they humans, non-human animals or machines. Here, we examine the role of a single variable, the difficulty of training, on the rate of learning. In many situations we find that there is a sweet spot in which training is neither too easy nor too hard, and where learning progresses most quickly. We derive conditions for this sweet spot for a broad class of learning algorithms in the context of binary classification tasks. For all of these stochastic gradient-descent based learning algorithms, we find that the optimal error rate for training is around 15.87% or, conversely, that the optimal training accuracy is about 85%. We demonstrate the efficacy of this ‘Eighty Five Percent Rule’ for artificial neural networks used in AI and biologically plausible neural networks thought to describe animal learning.
Introduction
The "Goldilocks zone" of learning—the idea that tasks should be neither too easy nor too difficult—is an intuitive concept in psychology and education. In machine learning, this is often operationalized through "Curriculum Learning," where models are presented with increasingly difficult examples. However, a precise mathematical definition of "optimal difficulty" has remained elusive. This research addresses this gap by deriving a formal rule for the optimal training error rate in stochastic gradient descent (SGD) systems.
Theoretical Framework
The researchers modeled learning as the improvement of a precision parameter ($\beta$) in a binary classification task. By analyzing the gradient of the error rate with respect to this precision, they identified the point of maximum information gain per training example.
The derivation shows that for any learning algorithm based on SGD with Gaussian noise, the optimal error rate is exactly $1 - \Phi(1) \approx 15.87\%$. This suggests that a training accuracy of approximately 85% is a fundamental mathematical constant for efficient learning in these systems.
Experimental Validation
To test the "Eighty Five Percent Rule," the authors conducted simulations across three distinct types of learners:
- Perceptrons: Single-layer neural networks using synthetic stimuli.
- Deep Neural Networks: Two-layer networks trained on binary versions of the MNIST handwritten digit dataset (e.g., distinguishing odd vs. even numbers).
- Biological Models: The Law and Gold model, which describes how monkeys learn to categorize the direction of moving dots in Random Dot Motion (RDM) tasks.
Key Results
The experiments yielded several critical insights:
- Superior Performance: Models trained at the 85% accuracy threshold achieved higher final test accuracy than those trained at 70% or 95%.
- Exponential Speedup: While training at a fixed difficulty leads to logarithmic growth in precision, "staircasing" difficulty to maintain an 85% accuracy level results in power-law growth, providing an exponential speedup in learning.
- Biological Alignment: The rule successfully predicted learning trajectories in biologically plausible models, suggesting that biological systems may have evolved to seek this optimal threshold.
Discussion and Conclusion
The 85% rule provides a theoretical link to the psychological state of "Flow"—the region of maximum learning gradient. While the rule is derived for binary tasks with Gaussian noise, its implications are broad. In environments with heavier-tailed noise (e.g., Cauchy noise), the optimal accuracy may drop to 75%, suggesting that higher error tolerance is beneficial in more unpredictable settings.
Future research will explore how this rule extends to multi-class classification and batch-based learning in modern deep learning architectures. For now, the 85% rule offers a powerful heuristic for educators and AI researchers alike: to maximize learning, keep the learner challenged but successful most of the time.
Summary updated