Bounded Rational Decision-Making in Feedforward Neural Networks

Felix Leibfried, Daniel A. BraunUAI 2016

Abstract

This paper introduces a principled framework for regularizing neural networks by treating them as "bounded rational" decision-makers. By applying information-theoretic constraints to weight updates, the authors derive regularization methods that prevent overfitting by limiting a network's informational resources. Evaluated on the MNIST benchmark, these methods achieve performance competitive with established techniques like Dropout and DropConnect, while offering a more formal foundation for network economy.

Introduction

In classical decision theory, a "fully rational" agent maximizes utility without regard for computational cost. However, in practical machine learning, information processing is not free. Neural networks often suffer from overfitting because their high capacity allows them to memorize training data noise rather than learning generalizable patterns. While traditional solutions like $L_2$ regularization or Dropout are effective, they are largely heuristic. This research seeks a formal, information-theoretic foundation for regularization: how can we constrain a network's informational resources to force the learning of only the most essential features?

Methodology

The authors utilize the framework of Bounded Rationality, specifically the Rate Distortion Principle, modeling a neural network as an information channel with a limited transmission rate. To minimize informational "costs," the network is encouraged to keep its specific responses to inputs close to its average response. Two primary implementations are proposed:

  1. Local Rate Distortion (LRDI): Each individual neuron acts as a bounded rational agent, maximizing global utility while minimizing the mutual information between its specific inputs and its firing state.
  2. Global Rate Distortion (GRDI): The entire network is treated as a single agent, minimizing the mutual information between the raw input and the final classification output.

Mathematically, this results in a modified backpropagation rule. The update includes a term penalizing neurons for deviating from their mean firing rate (tracked via an exponential moving average). This prevents weights from growing uncontrollably and forces the network to be "economical" with its information processing.

Trade-off analysis and learning curves for Grdi and Lrdi. The top and middle rows show the impact of the trade-off parameter β on utility and error, while the bottom row illustrates faster initial learning compared to unregularized models (Umax).

Key Insights

  • Regularization as "Economy": Regularization is viewed as a budget constraint. A bounded rational network is "lazy"—it will not change its behavior for a specific input unless the utility gain outweighs the informational cost.
  • The Moving Average Anchor: The derived update rules naturally incorporate a moving average of neuron activity, acting as a homeostatic anchor that prevents over-specialization to rare training examples.
  • Global vs. Local Robustness: The Global (GRDI) approach was found to be more robust and easier to tune than the Local (LRDI) approach, effectively regularizing the entire information pipeline.

Experimental Results

The researchers evaluated their approach using the MNIST benchmark across two setups: a permutation-invariant multilayer perceptron and a Convolutional Neural Network (CNN). The information-theoretic approach proved highly competitive with state-of-the-art heuristics.

Comparison of classification errors on permutation-invariant MNIST for various regularization methods and network widths. GRDI consistently outperforms Bayes by Backprop and remains competitive with Dropout across different capacities.

On the permutation-invariant MNIST task, GRDI achieved a test error of 1.11%, outperforming Bayes by Backprop (1.32%) and performing similarly to Dropout (1.01% - 1.28%). When applied to a CNN architecture, GRDI reached a 0.61% error rate, competitive with DropConnect (0.63%) and Dropout (0.59%). Furthermore, the regularization effect remained steady across different network widths (from 529 to 4096 neurons), demonstrating its ability to prevent overfitting even as model capacity increases.

Conclusion and Future Work

While the information-theoretic approach provides a robust alternative to heuristic regularization, challenges remain in automating the selection of the trade-off parameter $\beta$. Future work aims to extend these deterministic feedforward rules to stochastic environments and Reinforcement Learning, as well as scaling the approach to larger datasets like ImageNet.

Summary updated

BibTeX
@inproceedings{leibfried2016bounded,
  author = {Felix Leibfried and Daniel A. Braun},
  title = {{Bounded Rational Decision-Making in Feedforward Neural Networks}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2016},
}
Figure