Topographic Transformation as a Discrete Latent Variable

Nebojsa Jojic, Brendan J. FreyNeurIPS 1999 · OpenReview

The Challenge of Transformation Invariance

In computer vision and pattern recognition, a fundamental goal is for models to be "invariant" to simple topographic changes. For instance, if you shift a picture of a handwritten digit "7" three pixels to the left, it remains a "7."

Standard generative models, such as Gaussian Mixture Models (GMMs) or Factor Analysis, inherently struggle with this concept. Because they lack an intrinsic understanding of spatial movement, they attempt to model every possible shifted, rotated, or sheared version of an object as a completely unique pattern. This limitation results in "blurry" models that require massive amounts of training data to cover every possible variation. While some researchers have utilized linear approximations to handle tiny transformations (like tangent propagation), these methods break down as soon as the movement becomes significant, such as a large rotation or a long translation.

Transformations as Discrete Latent Variables

To address this, the authors propose a novel framework that builds transformation invariance directly into generative models by treating the transformation itself as a discrete latent variable (denoted as $\ell$).

Instead of trying to model a continuous, highly non-linear "transformation manifold," the authors approximate this manifold with a set of discrete points.

Graphical models illustrating the discrete transformation variable $\ell$ alongside latent variables and observations.

The generative process operates as follows:

  1. Latent Generation: The model first generates a "clean," centered latent image $z$.
  2. Transformation Selection: It then selects a discrete transformation $\ell$ from a predefined set.
  3. Application: The transformation is applied to $z$ using a sparse transformation matrix $G_\ell$. For example, an integer-pixel translation is simply represented as a permutation matrix.
  4. Observation: Finally, pixel noise is added to produce the observed image $x$.

This core idea can be integrated into several standard models:

  • Transformed Mixtures of Gaussians (TMG): Adds transformation invariance to standard clustering.
  • Transformed Component Analysis (TCA): A "transformed factor analysis" where the latent image is modeled by linear combinations of factors.
  • Mixtures of Transformed Component Analyzers (MTCA): A hybrid that models clusters, linear components, and transformations simultaneously.

To train these models, the authors extend the standard Expectation-Maximization (EM) algorithm. During the E-step, the model computes the "responsibility" of each transformation—essentially asking, "Given this observed image, which shift or shear most likely produced it?" Crucially, because the transformation matrices ($G_\ell$) are sparse, the TMG model achieves $O(N)$ linear time complexity (where $N$ is the number of pixels), a massive improvement over the $O(N^3)$ time required by standard Factor Analysis likelihood computations.

Experimental Validation

The authors evaluated their approach across three distinct datasets, demonstrating significant improvements in denoising, clustering, and recognition.

1. Image Denoising (SEM Images) Using 230 low-signal-to-noise images from a Scanning Electron Microscope, the TMG model produced significantly sharper mean images and lower variance compared to simple averaging. Standard averaging usually blurs structures due to spatial uncertainty, but TMG successfully aligns the images before averaging.

2. Clutter Suppression and Face Clustering The model was tested on video frames featuring people walking across cluttered backgrounds or changing poses.

Face clustering results demonstrating the model's ability to suppress clutter and extract sharp cluster means.

In these experiments, the model learned to treat the moving person as the primary signal and the static, cluttered background as noise. This effectively allowed the model to "auto-crop" the faces, resulting in sharp, distinct cluster means that standard models fail to capture.

3. Handwritten Digit Modeling The most dramatic results were seen on a dataset of 2,000 greyscale handwritten digits (8x8 pixels). When tasked with sorting the digits into 10 clusters without any labels (unsupervised learning):

  • Standard GMM: 53% error
  • Mixture of Factor Analyzers: 49% error
  • TMG (Proposed): 26% error

Furthermore, on a supervised digit recognition task, TCA achieved a 2.7% error rate, outperforming standard Factor Analysis (3.2%). By explicitly modeling the transformation, the latent variable $z$ is "un-warped" and released from the burden of representing movement, allowing it to focus entirely on the intrinsic shape of the digit.

Limitations and Future Directions

While highly effective, the framework does have limitations. The primary drawback is the curse of dimensionality: the number of discrete transformations grows exponentially with the degrees of freedom. Modeling translation (X and Y), rotation, scale, and shear simultaneously results in a massive set of discrete points. Additionally, because the model uses a discrete approximation of a continuous manifold, it may miss subtle sub-pixel variations.

For future work, the authors suggest exploring variational learning methods to speed up the algorithm and extending the framework to handle time-series data, such as tracking objects through video frames.

Summary updated

Abstract

Invariance to topographic transformations such as translation and shearing in an image has been successfully incorporated into feed(cid:173) forward mechanisms, e.g., "convolutional neural networks", "tan(cid:173) gent propagation". We describe a way to add transformation invari(cid:173) ance to a generative density model by approximating the nonlinear transformation manifold by a discrete set of transformations. An EM algorithm for the original model can be extended to the new model by computing expectations over the set of transformations. We show how to add a discrete transformation variable to Gaussian mixture modeling, factor analysis and mixtures of factor analysis. We give results on filtering microscopy images, face and facial pose clustering, and handwritten digit modeling and recognition.
BibTeX
@inproceedings{jojic1999topographic,
  author = {Nebojsa Jojic and Brendan J. Frey},
  title = {{Topographic Transformation as a Discrete Latent Variable}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1999},
  url = {https://openreview.net/forum?id=ry-UAIWObr},
}
Figure