Learning Mackey-Glass from 25 Examples, Plus or Minus 2

Mark Plutowski, Garrison W. Cottrell, Halbert WhiteNeurIPS 1993

TL;DR

This paper demonstrates that massive datasets are not strictly necessary to train neural networks on complex, chaotic systems. By employing "active exemplar selection"—specifically a method called $\Delta ISB$—the authors successfully trained a network to predict the Mackey-Glass time series using only about 25 carefully chosen data points. This approach achieved a 10x to 20x reduction in computational cost while maintaining or even improving generalization capabilities.

The Challenge of Active Learning

In the early 1990s, as neural networks gained traction, researchers faced a significant bottleneck: training on large datasets is computationally expensive. The standard "strawman" approach involved feeding the network every available piece of data. However, much of this data is often redundant or uninformative.

The authors address the challenge of active learning: how can a model intelligently select the most informative training examples from a large pool of candidates? They test this on the Mackey-Glass equation, a classic benchmark for chaotic time series prediction. The primary objective is to minimize the "Integrated Squared Bias" (ISB) using the smallest possible subset of data and the simplest possible network architecture.

The $\Delta ISB$ Algorithm

The core contribution of the paper is the $\Delta ISB$ algorithm. Instead of training on the entire dataset, the algorithm initializes with a tiny training set (just one example) and a small network. It then iterates through the following steps:

  1. Active Selection: The algorithm evaluates the entire candidate pool and selects the single example that would most significantly decrease the Integrated Squared Bias. Mathematically, this involves finding the data point where the individual error gradient is most highly correlated with the average error gradient of the entire dataset.
  2. Network Growth: If the network struggles to fit the current subset (i.e., training stalls), the algorithm automatically adds hidden units. This ensures the model's complexity scales appropriately with the difficulty of the selected data.
  3. Redundancy Check: An "exemplar spacing" parameter is used to reject new points that are too close to existing ones in both input and output space, preventing the model from wasting resources on redundant information.

The authors also compare $\Delta ISB$ to a simpler heuristic called "Maximum Error," which simply selects the point where the current model’s prediction is most inaccurate.

Datasets and Evaluation

The study utilizes the Mackey-Glass time series, a non-linear delay differential equation often used to model physiological control systems. It is renowned in the machine learning community for its chaotic nature, making it a challenging target for long-term prediction.

  • Data Set 1: 1000 time steps used for initial sampling and testing.
  • Data Set 2: 2000 time steps, starting 5000 steps after the first set, used to test the model's ability to generalize to the distant future of the chaotic system.
Time series prediction

Key Insights

  • Data Quality over Data Quantity: The most striking insight is that a chaotic system with a fractal dimension of 2.1 can be accurately captured with just ~25 points. This suggests that for many problems, models are over-trained on redundant data.
  • Gradient Correlation: The $\Delta ISB$ criterion offers a compelling heuristic interpretation: it selects the example that directs the network's weights in the direction that helps fix the error across the entire potential dataset.
  • Active Selection as Regularization: Surprisingly, networks trained on fewer, actively selected points often performed better on novel test data than those trained on the full dataset. By ignoring the noise of redundant points, the active selection process acts as a form of regularization, preventing the model from overfitting to the density of the candidate set.
  • The Power of Simplicity: While $\Delta ISB$ is mathematically rigorous, the simpler "Maximum Error" heuristic performed nearly as well. This indicates that simply identifying where the model is most confused is a highly effective and computationally cheaper strategy for data selection.

Results and Efficiency

  • Computational Efficiency: $\Delta ISB$ required an order of magnitude fewer floating-point operations (FLOPs) compared to the standard approach. For a target root mean square error (RMSE) of 0.01, the strawman method required 4,740 million multiplies, while $\Delta ISB$ required only 219 million.
Cost vs Fit comparison
  • Conciseness: The algorithm consistently converged on using 25 examples (±2) and 3 to 6 hidden units to solve the problem effectively.
  • Generalization: In 9 out of 25 runs, the $\Delta ISB$ networks performed better on a completely different part of the time series (Data Set 2) than they did on their own candidate pool—a feat the "all-data" approach never achieved.

Limitations and Future Directions

  • Approximation Sensitivity: The algorithm relies on approximating a Hessian-like matrix ($H$). While replacing $H$ with an identity matrix worked well for this specific task, more complex problems might necessitate more precise and computationally expensive calculations.
  • Heuristic Justification: Although the "Maximum Error" method performed excellently, it lacks the rigorous analytical justification of $\Delta ISB$. Future work should investigate why this simpler heuristic proves so robust.
  • Parameter Tuning: Parameters like "exemplar spacing" and the "fit requirement" ($F_N$) require manual tuning, which could be challenging for datasets where the scale of the input/output space is not well understood.

Summary updated

BibTeX
@inproceedings{plutowski1993learning,
  author = {Mark Plutowski and Garrison W. Cottrell and Halbert White},
  title = {{Learning Mackey-Glass from 25 Examples, Plus or Minus 2}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
}
Figure