Adapting Bias by Gradient Descent: An Incremental Version of Delta-Bar-Delta

Richard S. SuttonAAAI 1992 · OpenReview

Abstract

Appropriate bias is widely viewed as the key to efficient learning and generalization. I present a new algorithm, the Incremental Delta-Bar-Delta (IDBD) algorithm, for the learning of appropriate biases based on previous learning experience. The IDBD algorithm is developed for the case of a simple, linear learning system--the LMS or delta rule with a separate learning-rate parameter for each input. The IDBD algorithm adjusts the learning-rate parameters, which are an important form of bias for this system. Because bias in this approach is adapted based on previous learning experience, the appropriate test beds are drifting or non-stationary learning tasks. For particular tasks of this type, I show that the IDBD algorithm performs better than ordinary LMS and in fact finds the optimal learning rates. The IDBD algorithm extends and improves over prior work by Jacobs and by me in that it is fully incremental and has only a single free parameter. This paper also extends previous work by presenting a derivation of the IDBD algorithm as gradient descent in the space of learning-rate parameters. Finally, I offer a novel interpretation of the IDBD algorithm as an incremental form of hold-one-out cross validation.

Introduction

In machine learning, "bias" refers to the prior preferences a learner has for certain hypotheses over others. Humans learn incredibly fast because we bring the right biases to a problem—we intuitively know which features of our environment usually matter. Most traditional algorithms, however, use a single, fixed learning rate ($\alpha$) for all inputs. This is often inefficient for two reasons: it fails to ignore irrelevant noise, and it struggles with non-stationary tasks where the underlying rules of the environment change over time.

Richard Sutton’s 1992 paper introduces the Incremental Delta-Bar-Delta (IDBD) algorithm to solve these issues. IDBD is a meta-learning method that automatically tunes individual learning rates for every input in a linear system, allowing the model to "learn how to learn" from its own experience.

The IDBD Algorithm

The core innovation of IDBD is treating the learning rate itself as a parameter to be optimized via gradient descent. The algorithm builds on the Least-Mean-Square (LMS) rule with several key technical features:

  • Per-Input Learning Rates: Every input $x_i$ is assigned its own learning rate $\alpha_i$.
  • Exponential Parameterization: Learning rates are updated via a meta-parameter $\beta_i$, where $\alpha_i = e^{\beta_i}$. This ensures that $\alpha_i$ remains positive and changes geometrically.
  • The Memory Trace ($h_i$): The algorithm maintains a decaying trace of recent weight changes. If a current weight change correlates positively with the trace, the meta-learning rate increases $\beta_i$, signaling that the input is consistently useful.
  • Fully Incremental: Unlike previous batch methods, IDBD processes examples one by one, making it suitable for real-time, online learning.

Experimental Results

Sutton evaluated IDBD using synthetic tracking tasks involving 20 real-valued inputs. In these tasks, only 5 inputs were "relevant" to the target output, while 15 were "irrelevant" noise. To test adaptability, the signs of the relevant weights were flipped periodically, forcing the model to constantly re-learn.

Automatic Feature Selection

One of the most powerful aspects of IDBD is its ability to perform automatic feature selection. By observing the correlation of weight updates, the algorithm identifies which inputs contribute to the signal and which do not.

Evolution of learning rates for relevant and irrelevant inputs over time.

As shown in the figure above, the learning rates for relevant inputs increase and stabilize, while the rates for irrelevant inputs are driven toward zero. This effectively "shuts off" the noise, allowing the model to focus exclusively on the data that matters.

Performance and Robustness

In drifting target experiments, IDBD significantly outperformed standard LMS. While the best possible fixed-rate LMS algorithm achieved a mean squared error of approximately 3.5, IDBD reached an error of 1.5.

Comparison of asymptotic error between LMS and IDBD across different parameter settings.

Furthermore, IDBD proved to be more robust. As illustrated in the performance comparison, IDBD maintains a low asymptotic error across a wide range of its meta-parameter $\theta$, whereas LMS is highly sensitive to the choice of its global learning rate $\alpha$.

Conclusion

IDBD provides a formal framework for meta-learning as stochastic gradient descent in the space of learning-rate parameters. By interpreting the algorithm as an incremental form of "hold-one-out" cross-validation, Sutton demonstrates how a system can validate its internal biases in real-time. While the paper focuses on linear units, the principles of adaptive bias and per-input learning rates remain foundational concepts in modern optimization and neural network research.

Summary updated

Abstract

Appropriate bias is widely viewed as the key to efficient learning and generalization. I present a new algorithm, the Incremental Delta-Bar-Delta (IDBD) algorithm, for the learning of appropriate biases based on previous learning experience. The IDBD algorithm is developed for the case of a simple, linear learning system--the LMS or delta rule with a separate learning-rate parameter for each input. The IDBD algorithm adjusts the learning-rate parameters, which are an important form of bias for this system. Because bias in this approach is adapted based on previous learning experience, the appropriate test beds are drifting or non-stationary learning tasks. For particular tasks of this type, I show that the IDBD algorithm performs better than ordinary LMS and in fact finds the optimal learning rates. The IDBD algorithm extends and improves over prior work by Jacobs and by me in that it is fully incremental and has only a single free parameter. This paper also extends previous work by presenting a derivation of the IDBD algorithm as gradient descent in the space of learning-rate parameters. Finally, I offer a novel interpretation of the IDBD algorithm as an incremental form of hold-one-out cross validation.
BibTeX
@inproceedings{sutton1992adapting,
  author = {Richard S. Sutton},
  title = {{Adapting Bias by Gradient Descent: An Incremental Version of Delta-Bar-Delta}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1992},
  url = {https://openreview.net/forum?id=ByZQKxZOZH},
}
Figure