The Eighty Five Percent Rule for optimal learning

Robert C. Wilson, Amitai Shenhav, Mark A. Straccia, Jonathan D. Cohen

Abstract

Researchers and educators have long wrestled with the question of how best to teach their clients be they humans, non-human animals or machines. Here, we examine the role of a single variable, the difficulty of training, on the rate of learning. In many situations we find that there is a sweet spot in which training is neither too easy nor too hard, and where learning progresses most quickly. We derive conditions for this sweet spot for a broad class of learning algorithms in the context of binary classification tasks. For all of these stochastic gradient-descent based learning algorithms, we find that the optimal error rate for training is around 15.87% or, conversely, that the optimal training accuracy is about 85%. We demonstrate the efficacy of this ‘Eighty Five Percent Rule’ for artificial neural networks used in AI and biologically plausible neural networks thought to describe animal learning.

Introduction

The "Goldilocks zone" of learning—the idea that tasks should be neither too easy nor too difficult—is an intuitive concept in psychology and education. In machine learning, this is often operationalized through "Curriculum Learning," where models are presented with increasingly difficult examples. However, a precise mathematical definition of "optimal difficulty" has remained elusive. This research addresses this gap by deriving a formal rule for the optimal training error rate in stochastic gradient descent (SGD) systems.

Theoretical Framework

The researchers modeled learning as the improvement of a precision parameter ($\beta$) in a binary classification task. By analyzing the gradient of the error rate with respect to this precision, they identified the point of maximum information gain per training example.

Figure 1: Conceptual framework for the 85% rule. (a) Distribution of the decision variable h before and after learning. (b) Relationship between error rate (ER) and task difficulty. (c-d) The learning rate as a function of difficulty and error rate, showing a clear peak.

The derivation shows that for any learning algorithm based on SGD with Gaussian noise, the optimal error rate is exactly $1 - \Phi(1) \approx 15.87\%$. This suggests that a training accuracy of approximately 85% is a fundamental mathematical constant for efficient learning in these systems.

Experimental Validation

To test the "Eighty Five Percent Rule," the authors conducted simulations across three distinct types of learners:

  1. Perceptrons: Single-layer neural networks using synthetic stimuli.
  2. Deep Neural Networks: Two-layer networks trained on binary versions of the MNIST handwritten digit dataset (e.g., distinguishing odd vs. even numbers).
  3. Biological Models: The Law and Gold model, which describes how monkeys learn to categorize the direction of moving dots in Random Dot Motion (RDM) tasks.
Figure 2: Detail of the learning rate peak at an error rate of 0.1587, the mathematical basis for the 85% rule.

Key Results

The experiments yielded several critical insights:

  • Superior Performance: Models trained at the 85% accuracy threshold achieved higher final test accuracy than those trained at 70% or 95%.
  • Exponential Speedup: While training at a fixed difficulty leads to logarithmic growth in precision, "staircasing" difficulty to maintain an 85% accuracy level results in power-law growth, providing an exponential speedup in learning.
  • Biological Alignment: The rule successfully predicted learning trajectories in biologically plausible models, suggesting that biological systems may have evolved to seek this optimal threshold.

Discussion and Conclusion

The 85% rule provides a theoretical link to the psychological state of "Flow"—the region of maximum learning gradient. While the rule is derived for binary tasks with Gaussian noise, its implications are broad. In environments with heavier-tailed noise (e.g., Cauchy noise), the optimal accuracy may drop to 75%, suggesting that higher error tolerance is beneficial in more unpredictable settings.

Future research will explore how this rule extends to multi-class classification and batch-based learning in modern deep learning architectures. For now, the 85% rule offers a powerful heuristic for educators and AI researchers alike: to maximize learning, keep the learner challenged but successful most of the time.

Summary updated

Abstract

Abstract Researchers and educators have long wrestled with the question of how best to teach their clients be they humans, non-human animals or machines. Here, we examine the role of a single variable, the difficulty of training, on the rate of learning. In many situations we find that there is a sweet spot in which training is neither too easy nor too hard, and where learning progresses most quickly. We derive conditions for this sweet spot for a broad class of learning algorithms in the context of binary classification tasks. For all of these stochastic gradient-descent based learning algorithms, we find that the optimal error rate for training is around 15.87% or, conversely, that the optimal training accuracy is about 85%. We demonstrate the efficacy of this ‘Eighty Five Percent Rule’ for artificial neural networks used in AI and biologically plausible neural networks thought to describe animal learning.
BibTeX
@inproceedings{wilson2019eighty,
  author = {Robert C. Wilson and Amitai Shenhav and Mark A. Straccia and Jonathan D. Cohen},
  title = {{The Eighty Five Percent Rule for optimal learning}},
  booktitle = {Nature Communications},
  year = {2019},
  doi = {10.1038/s41467-019-12552-4},
}
Figure