Unsupervised Classifiers, Mutual Information and 'Phantom Targets'

John S. Bridle, Anthony J. R. Heading, David J. C. MacKayNeurIPS 1991

Overview

This paper introduces a method to transform standard supervised classifiers into unsupervised data analyzers by maximizing the Mutual Information (MI) between inputs and outputs. The authors propose a "firm but fair" objective function that forces a neural network to be confident in its individual predictions while ensuring it utilizes all available categories equally across the dataset.

The Problem: Bridging Supervised and Unsupervised Learning

In the early 1990s, the boundary between supervised learning (learning from labeled data) and unsupervised learning (discovering patterns in raw data) was a major focus of neural network research. The authors sought to build a bridge between these two paradigms.

Specifically, they asked: Can we take a standard classifier architecture—designed to predict labels—and "bootstrap" it to discover clusters in unlabeled data?

While techniques like Principal Component Analysis (PCA) or Vector Quantization (VQ) existed, they often relied on minimizing reconstruction error. The authors aimed for a more general, information-theoretic criterion applicable to any network producing probability-like outputs (e.g., via a SoftMax layer).

Methods

The paper proposes two main approaches, with the second being the primary contribution:

  1. The Phantom Target Rule: For specific Gaussian classifiers, the authors demonstrate that doubling the network's own output ($t = 2y$) and using that as a "target" for backpropagation is mathematically equivalent to performing a Maximum Likelihood fit of a Gaussian Mixture Model.
  2. The Mutual Information (MI) Criterion: This is the more general "Firm but Fair" approach. The authors define an objective function based on the mutual information $I(c; x)$ between the input $x$ and the predicted class $c$.

Mathematically, this objective is decomposed into two entropy terms:

  • Fairness ($H(\bar{y})$): The entropy of the average output across the whole dataset. Maximizing this ensures the network doesn't simply assign one class to every single input; it encourages an even distribution of patterns across all available classes.
  • Firmness ($H(y)$): The average entropy of individual outputs. Minimizing this (or subtracting it in the objective) forces the network to be "decisive" or "firm." For any given input, it should select one class with high confidence rather than remaining lukewarm across several.

Training requires a two-pass approach: one pass through the data to calculate the average class activities, and a second pass to backpropagate the derivatives of the MI objective.

Experimental Setup

To illustrate the approach, the authors employ a small-scale synthetic dataset:

  • 2D Isotropic Gaussian: 30 data points drawn from a single Gaussian distribution in a two-dimensional space.
  • Reference Points: 10 "class" centers (reference points) were initialized near the center of the data cluster to observe their movement during unsupervised training.

Key Results

The experiments successfully demonstrated the viability of the MI criterion for discovering structure in unlabeled data.

Composite results showing M.I. convergence, Firm/Fair components, reference point tracks, and final decision regions.
  • Objective Convergence: The MI criterion (normalized by its maximum possible value) increased steadily over 100 iterations, demonstrating that the gradient descent approach successfully optimizes the "firm but fair" trade-off.
  • Spatial Partitioning: In the 2D experiment, the 10 reference points—initially clumped together—moved outward to form a ring around the data. This effectively partitioned the single Gaussian cloud into 10 Voronoi-like regions, each containing a roughly equal number of points.
  • Decision Boundaries: The resulting decision regions showed that the network learned to grow "firmer" boundaries as the reference points moved apart, solidifying its unsupervised classifications.

Key Insights

  • The "Firm but Fair" Mnemonic: This is the standout conceptual takeaway. A good unsupervised classifier should be "firm" (confident in its specific assignments) but "fair" (utilizing its entire "vocabulary" of classes across the dataset).
  • Information as a Bridge: By framing the problem as maximizing the information the output conveys about the input, the authors provide an objective that is independent of the specific network architecture.
  • Simplicity as Regularization: The authors note that for complex, unregularized models, this objective could lead to implausible hard partitions. However, for simpler models (like the Gaussian classifier shown), the inherent constraints of the model prevent it from creating contorted regions, leading to more natural-looking clusters.

Limitations and Future Work

  • Computational Overhead: The requirement for two passes over the training set makes this more computationally expensive than standard supervised backpropagation.
  • The "Naturalness" Problem: The authors admit that "natural" clusters are ill-defined. In a highly flexible (poorly regularized) model, the network might find a mathematically optimal "firm and fair" solution that doesn't correspond to meaningful clusters in the data.
  • Weighting Components: The paper suggests that future research might need to experiment with different weightings for the "firmness" versus "fairness" components to achieve better results on specific data types.
  • Regularization: The authors emphasize that this criterion should be paired with objective techniques for regularizing networks to ensure the discovered partitions remain simple and useful.

Summary updated

BibTeX
@inproceedings{bridle1991unsupervised,
  author = {John S. Bridle and Anthony J. R. Heading and David J. C. MacKay},
  title = {{Unsupervised Classifiers, Mutual Information and 'Phantom Targets'}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1991},
}
Figure