Compact Part-Based Image Representations: Extremal Competition and Overgeneralization

Marc Goessling and Yali AmitICLR 2015 · arXiv · OpenReview

Introduction

While deep neural networks are incredibly powerful at classification, the internal features they learn are often high-dimensional, redundant, and difficult for humans to interpret. For instance, a network might recognize the letter "T" using dozens of overlapping, fuzzy features rather than the two simple, distinct bars (horizontal and vertical) that a human would naturally use.

This paper addresses a fundamental flaw in current models like Restricted Boltzmann Machines (RBMs): a "lack of competition." Because these models typically sum up expert opinions (often in the log-odds space), multiple experts frequently end up learning the exact same structures. To solve this, the authors introduce a novel way to compose "expert" models to represent binary images using a "max-minus-min" rule. By forcing experts to compete for pixels rather than averaging their opinions, the model learns sparse, semantically meaningful parts from just a few examples.

The Shift to Extremal Composition

The core contribution of this work is a paradigm shift in how individual expert models are combined, moving from "cooperative" to "extremal" composition.

  1. The Max-Minus-Min Rule: In standard cooperative models, if two experts both believe a pixel should be "on," their combined probability increases. Under the proposed extremal rule, the model only listens to the most extreme opinion. If one expert is highly confident a pixel is black and another is "not sure" (abstaining), the model simply ignores the unsure expert. This prevents redundancy; once one expert explains a part of the data, there is no incentive for another expert to mimic it.
  2. Likelihood Matching Pursuit: To determine which experts are present in a given image, the authors employ a sequential inference procedure. They greedily add the expert that most improves the overall likelihood, a process conceptually similar to "Matching Pursuit" in signal processing.
  3. Sequential Initialization: Instead of starting with random noise, the model initializes with a single, oversimplified global template. It then identifies where this template fails to explain new data and "corrects" it by spawning new experts specifically tailored to cover those discrepancies.
Comparison of composition functions and their visual effects on combining two image parts. The top plots show the mathematical behavior of various rules, while the bottom row demonstrates how the proposed maximum-minus-minimum rule perfectly preserves overlapping parts without washing them out.

Experimental Setup

The authors evaluated their extremal composition approach on two primary types of data:

  • Synthetic Binary Images: A 6x6 grid where quadrants are randomly turned on or off. This provides a clear ground truth to test whether the model can successfully recover the 8 underlying "parts" (4 quadrants × 2 polarities).
  • TiCC Handwritten Characters: A dataset of handwritten letters (56x56 pixels). The model was tasked with learning the fundamental strokes of letters like 'A', 'B', and 'T' while accounting for natural shifts and rotations.

Key Findings and Insights

Competition Breeds Sparsity

By utilizing the "max" or "max-minus-min" functions, the responsibility for explaining a single pixel is never split between experts. This "winner-take-all" dynamic naturally leads to models where each expert represents a distinct, non-overlapping part of the image.

Abstention is a Virtue

The authors define a specific probability value (usually 0.5 for symmetric models) that represents an expert "abstaining" from a vote. The extremal rule makes it highly attractive for experts to stay in this neutral state unless they have a unique, confident contribution to make, further driving sparsity.

Superior Disentanglement and Efficiency

On the synthetic dataset, the proposed model almost perfectly recovered the original quadrant experts. In stark contrast, standard RBMs and Denoising Autoencoders (DAEs) produced "mushy" templates that were entangled mixtures of multiple quadrants. Furthermore, the max-minus-min model achieved significantly lower cross-entropy reconstruction error than RBMs or DAEs, particularly in low-data regimes (e.g., under 100 training examples).

Semantic Letter Parts

When applied to real-world handwritten characters, the model successfully extracted highly interpretable components. For the letter 'T', the model converged to exactly two experts: a horizontal bar and a vertical bar. By combining these learned parts with a Gaussian distribution to model their relative spatial positions, the framework could generate realistic and varied samples of the letter.

Limitations and Future Directions

While highly effective for specific domains, the current framework has limitations. Most notably, it is strictly designed for binary (0/1) data. While this is a natural fit for black-and-white ink, text, or silhouettes, it does not directly apply to natural grayscale or color images. Additionally, while the model excels at simple "part-based" objects like letters, it remains to be seen how well this extremal competition scales to highly complex scenes with hundreds of overlapping objects.

Future work will focus on extending these composition rules to handle variances in real-valued data and developing mechanisms to ensure smooth transitions when moving from one expert's spatial influence to another.

Summary updated

BibTeX
@inproceedings{amit2015compact,
  author = {Marc Goessling and Yali Amit},
  title = {{Compact Part-Based Image Representations: Extremal Competition and Overgeneralization}},
  booktitle = {International Conference on Learning Representations},
  year = {2015},
  url = {http://arxiv.org/abs/1412.3708},
}

Loading related content...

Figure