Gated Softmax Classification

Roland Memisevic, Christopher Zach, Geoffrey E. Hinton, Marc Pollefeys

Introduction

Standard classification models, such as logistic regression, operate on the assumption that classes can be separated by simple hyperplanes. However, real-world data often contains complex, unlabeled variations—what the authors refer to as "styles." For example, a handwritten "7" might look drastically different depending on whether it is italicized or written with a thick pen.

While deep neural networks address this by learning hierarchies of features, they can be difficult to train and interpret. To bridge this gap, this paper introduces the Gated Softmax Model (GSM), a probabilistic classifier that uses binary latent variables to capture these hidden styles. By factorizing a three-dimensional weight tensor, the model efficiently learns task-specific invariances and achieves classification performance competitive with Deep Belief Networks (DBNs) and Support Vector Machines (SVMs), all while maintaining a "shallow" architecture.

Methods: The Log-Bilinear Model

The core of the paper is the Log-Bilinear Model. Instead of a standard linear score function ($w^T x$), the authors introduce a vector of binary latent variables $h$. The score for a class $y$ is defined as:

$$s_y(x, h) = h^T W_y x$$

This creates a multiplicative interaction between the input $x$ and the hidden style $h$.

Architecture of the Gated Softmax Model showing standard and factorized 3-way interactions.

Key Innovations

  • Tractable Marginalization: Even though there are $2^K$ possible combinations of $K$ binary style features, the authors utilize a mathematical "re-arrangement" trick that allows them to compute the exact probability of a class in closed form. This avoids the need for expensive sampling or approximations common in other latent variable models.
  • Tensor Factorization: To prevent the number of parameters from exploding and to encourage "feature sharing," the authors factorize the 3D weight tensor $W$ into three smaller matrices ($W^x, W^y, W^h$). This allows the model to learn a dictionary of "filters" ($W^x$) that are combined based on the detected style and the target class.
  • RBM Initialization: Because the objective function is non-linear and prone to local optima, the authors initialize the model using class-specific Restricted Boltzmann Machines (RBMs), a technique reminiscent of the pre-training used in deep learning.

Key Insights

Style as a Gate: The latent variables act as "gates" that turn on or off different linear classifiers. The final model is essentially an ensemble of $2^K$ different logistic regressors that share weights.

Automatic Invariance: By factorizing the tensor, the model doesn't just learn features; it learns invariances. For the rotation task, the model automatically learned filters that resemble circular or radial Fourier components, effectively "undoing" the rotation to identify the digit.

Learned filters resembling circular or radial Fourier components for the rotation task.

The Power of Three-Way Interactions: The authors argue that the "magic" comes from the three-way interaction between inputs, hidden variables, and labels. This allows the model to be "shallow" (one hidden layer) yet capture dependencies that usually require deep architectures.

Datasets and Results

The model was evaluated on a deep learning benchmark consisting of 8 challenging image classification tasks, including distinguishing rectangles, identifying convex shapes, and classifying MNIST variations (rotated digits, random backgrounds, etc.).

The Gated Softmax Model proved to be a top-tier performer across the benchmark:

  • State-of-the-Art on Rectangles: On the basic rectangles task, the GSM achieved a 0.83% error rate, vastly outperforming SVMs (2.15%) and Deep Belief Nets (2.60%). An unfactored version did even better at 0.56%.
  • Competitive on Rotation: On the difficult mnist-rot dataset, the model achieved 11.75% error, which was competitive with the best deep models of the time.
  • General Performance: The GSM was among the best performers (within 0.01 tolerance) in 3 out of the 8 benchmark cases and remained competitive in the rest, proving that a well-designed shallow probabilistic model can hold its own against deep networks.

Limitations and Future Work

Despite its strong performance, the model has a few limitations. Its non-linear objective means it is highly sensitive to starting conditions; without RBM-based pre-training, standard conjugate gradient solvers often get stuck in poor local optima. Additionally, unlike deep networks, the GSM does not learn a multi-layer feature hierarchy, which may limit its performance on extremely high-level abstract reasoning tasks.

For future work, the authors suggest exploring a fully Bayesian treatment of the parameters, adding sparsity constraints to the hidden units, and extending the model to handle structured prediction tasks like Conditional Random Fields.

Summary updated

Abstract

We describe a log-bilinear" model that computes class probabilities by combining an input vector multiplicatively with a vector of binary latent variables. Even though the latent variables can take on exponentially many possible combinations of values, we can efficiently compute the exact probability of each class by marginalizing over the latent variables. This makes it possible to get the exact gradient of the log likelihood. The bilinear score-functions are defined using a three-dimensional weight tensor, and we show that factorizing this tensor allows the model to encode invariances inherent in a task by learning a dictionary of invariant basis functions. Experiments on a set of benchmark problems show that this fully probabilistic model can achieve classification performance that is competitive with (kernel) SVMs, backpropagation, and deep belief nets."
BibTeX
@inproceedings{memisevic2010gated,
  author = {Roland Memisevic and Christopher Zach and Geoffrey E. Hinton and Marc Pollefeys},
  title = {{Gated Softmax Classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
}
Figure