Unification of Information Maximization and Minimization

Ryotaro KamimuraNeurIPS 1996

This paper, published in the mid-90s by Ryotaro Kamimura, explores a fundamental tension in neural network training: the balance between making internal representations clear (Information Maximization) and making the network generalize well by keeping it simple (Information Minimization). To achieve both simultaneously, the author proposes a unified framework called the "Information Controller."

The "Black Box" Problem

In the mid-90s, neural networks were frequently criticized as opaque "black boxes." Researchers utilized two main information-theoretic strategies to address this, but they often seemed contradictory:

  1. Information Maximization (Infomax): This approach aims to make hidden unit activities as distinct and informative as possible. While this helps in "feature detection" and understanding what the network has learned, it can lead to overly complex weight structures.
  2. Information Minimization (Infomin): This strategy aims to reduce the complexity of the network, similar to weight decay or pruning. It improves generalization and prevents over-training, but it can result in "blurry" or hard-to-interpret internal representations.

The primary challenge was the mathematical difficulty of maximizing and minimizing the same information metric simultaneously. Kamimura’s framework bridges this gap by unifying these two goals into a single training paradigm.

The Information Controller

The core innovation of the paper is the Information Controller, which resolves the contradiction by splitting information management into two distinct levels: collective and individual.

Information Controller Architecture
  • Collective Level (Maximization): At this level, the controller evaluates the hidden layer as a whole. It uses a normalized output (akin to a softmax function) to ensure that for any given input, the network tries to activate a specific, distinct pattern of hidden units. Ideally, it pushes the network toward a state where only one or a few hidden units are "on" for a specific feature, maximizing the information content of the layer's activity.
  • Individual Level (Minimization): Conversely, the controller looks at individual input-to-hidden connections. It treats the strength of a connection as a probability of "firing." By minimizing this information, the network is forced to push as many weights as possible toward zero, effectively pruning unnecessary connections.

By separating these objectives, the author avoids mathematical contradiction. The network is trained using a combined cost function that includes standard cross-entropy (for accuracy) alongside these two information-theoretic terms.

Experimental Setup and Key Insights

The method was evaluated using an artificial language task involving syllabification (segmenting strings of characters into syllables based on the "maximum onset principle") and well-formedness inference (determining if a string follows specific rules). The experiments utilized training sets of 200 and 300 patterns.

The dual approach yielded several critical insights:

  • Level Separation: Defining "information" differently for activities versus weights allows the network to be pulled in two beneficial directions at once.
  • Automatic Pruning: Individual minimization acts as a sophisticated version of weight decay. It doesn't just shrink weights; it effectively shuts off entire hidden units that aren't contributing to the collective information.
  • Interpretable Sparsity: In the paper's results, a network initialized with 10 hidden units was automatically reduced to just 2 functional units. These remaining units could be easily mapped to specific linguistic rules, transforming the black box into a transparent, rule-based model.

Superior Generalization Results

The Information Controller was benchmarked against standard backpropagation, Weight Decay, and Weight Elimination.

Generalization Errors Comparison

As demonstrated in the 300-pattern test, the Information Controller significantly outperformed alternative methods:

  • Lower Error Rates: It achieved an impressive error rate of 0.008, compared to Standard learning (0.024) and Weight Decay (0.012).
  • Reduced RMS Error: The Root Mean Square (RMS) error for the Information Controller was 0.072, the lowest among all tested methods.

Limitations and Future Work

While groundbreaking for its time, the approach has notable limitations. Implementing the Information Controller is computationally more involved than standard methods, requiring the calculation and differentiation of entropy-based functions during every training step. Furthermore, the method introduces several new hyperparameters ($\beta, \mu, \eta$) without providing a systematic way to tune them for new problems. Finally, while successful on smaller artificial language tasks, the paper leaves open the question of how this dual-information approach would scale to the massive, multi-layer architectures that dominate modern Deep Learning.

Summary updated

BibTeX
@inproceedings{kamimura1996unification,
  author = {Ryotaro Kamimura},
  title = {{Unification of Information Maximization and Minimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1996},
}
Figure