Unsupervised Learning of Mixtures of Multiple Causes in Binary Data

Eric SaundNeurIPS 1993

Introduction

Traditional unsupervised learning techniques, such as k-means or standard mixture models, typically operate on a "single-cause" assumption: every data point is generated by exactly one cluster center. However, real-world data often involves multiple independent processes occurring simultaneously. For instance, if two different objects appear in the same image, a single-cause model would require a unique cluster for every possible combination of those objects, leading to a combinatorial explosion.

While some neural networks (like PCA or Boltzmann Machines) allow for multiple active hidden units, they often produce unintuitive representations. Their standard "weighted sum plus sigmoid" mixing functions allow errors from one unit to be canceled out by another. Without sufficient pressure, hidden units fail to specialize in specific, independent features of the data. To address this, the authors introduce a "multiple-cause" unsupervised learning model specifically designed for binary data. By utilizing a novel mixing function that allows hidden units to express "don't-know" states, the algorithm successfully decomposes complex patterns into their independent constituent parts.

The Multiple Cause Mixture Model

The proposed model is tailored for binary data, represented as 1 and -1. The core innovation lies in how the model handles weights and mixing:

  • Weight Representation: The weights ($c_{j,k}$) are designed such that their sign indicates whether a hidden unit predicts a 1 or -1, while the magnitude (capped at 1) represents the strength of that belief. Crucially, a weight of 0 represents a "don't-know" or "don't-care" state.
  • Nonlinear Mixing Function: Instead of a simple linear sum, the model employs a specialized nonlinear mixing function. If multiple hidden units disagree strongly on a pixel, the result is an "uncertain" prediction that incurs a penalty during learning. This mechanism forces units to only take responsibility for dimensions they can reliably predict, avoiding the error cancellation seen in standard neural networks.
  • Learning Process: The model optimizes an objective function equivalent to log-likelihood using gradient ascent, simultaneously adjusting both the hidden unit activities (recognition) and the weights (learning).

Datasets and Combinatorial Efficiency

The model was evaluated on three primary types of data:

  1. Synthetic "Two-Process" Data: 121-dimensional vectors representing two independent rectangles (one on the left, one on the right) moving to different positions.
  2. Noisy Synthetic Data: The same rectangle data augmented with 10% "bit-flip" noise to test robustness.
  3. Printed Characters: 21x21 pixel images (441-dimensional binary vectors) of 26 lowercase English characters.
Combinations of the synthetic two-process rectangle dataset.

The synthetic dataset perfectly illustrates the model's combinatorial efficiency. A traditional single-cause model requires 9 distinct clusters to represent all possible combinations of the rectangle positions. In contrast, the multiple-cause model needs only 6, as it learns to represent the underlying independent processes rather than their exhaustive combinations.

Key Insights and Results

The model's architecture leads to several significant advantages in feature discovery and representation:

  • The Power of "Don't Care": By allowing weights to be zero, the model can ignore irrelevant parts of the data vector. For example, a hidden unit can represent a "vertical stroke" in a character without having to make a guess about the rest of the image.
  • Coherent Feature Discovery: On the printed character dataset, the model did not merely learn rigid templates for each letter. With $K=14$ hidden units, it successfully discovered localized, sub-letter features such as ascenders (the top of a 'b' or 'd'), descenders (the tail of a 'p' or 'q'), and circular loops. This contrasts sharply with PCA, which tends to produce smeared, global components.
  • Noise Reduction: The model demonstrated strong robustness to noise. When presented with a "dirty" image containing 10% flipped bits, the model's reconstruction was significantly cleaner than the input. It effectively denoised the data by mapping it back to the learned causal structure.

Limitations and Future Work

While the multiple-cause model presents a strong step forward for unsupervised feature discovery, it has limitations. The current formulation (the "WRITE-WHITE-AND-BLACK" model) is strictly constrained to binary data, and extending it to continuous or multi-valued data remains unaddressed. Additionally, the mixing function used is a computationally tractable approximation; more mathematically rigorous functions might exist. Finally, the experiments were limited to relatively small 21x21 grids, leaving questions about how the gradient ascent approach scales to high-resolution images or massive datasets.

Summary updated

BibTeX
@inproceedings{saund1993unsupervised,
  author = {Eric Saund},
  title = {{Unsupervised Learning of Mixtures of Multiple Causes in Binary Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
}
Figure