Learning Lie Groups for Invariant Visual Perception

Rajesh P. N. Rao, Daniel L. RudermanNeurIPS 1998 · OpenReview

Introduction: The Challenge of Visual Invariance

A core challenge in both biological and machine vision is visual invariance: the ability to recognize an object regardless of whether it has been shifted, rotated, or scaled. Historically, researchers have used first-order Taylor series expansions to model these visual changes. While mathematically simple, these linear models are "short-sighted"—they only work for infinitesimal transformations. If an object moves significantly, the linear approximation breaks down.

In this paper, the authors introduce a Bayesian framework for learning the "grammar" of visual transformations directly from image data. By leveraging Lie group theory and a matrix-exponential generative model, they demonstrate an unsupervised algorithm that learns transformation operators from tiny movements and successfully applies them to handle large, macroscopic changes in an object's appearance.

A Continuous Model: Lie Groups and Matrix Exponentials

To overcome the limitations of linear approximations, the authors turn to Lie Group theory, which provides a formal way to describe continuous transformations.

Instead of a simple linear shift, they model a transformed image $I(x)$ using a matrix exponential: $$I(x) = e^{xG}I_0$$ Here, $I_0$ is the original image, $x$ is the magnitude of the transformation, and $G$ is the "generator" matrix (the Lie algebra element). The authors elegantly show that previous first-order models are simply the first two terms of the Taylor expansion of this exponential model. By using the full exponential, they move from local approximations to global ones.

The "What" vs. "Where" Architecture

The proposed architecture mirrors the primate visual system’s division of labor. The model splits the task into two cooperating networks: one estimates object identity ("what"), while a separate network estimates the transformation ("where/how").

Dual-network architecture separating object identity estimation from transformation estimation.

Unsupervised Bayesian Learning

The authors use a Maximum A Posteriori (MAP) approach to find the generator $G$. The algorithm alternates between two steps:

  1. Fast Estimation: Estimating the transformation amount ($x$) for a specific pair of images.
  2. Slow Adaptation: Updating the global generator matrix ($G$) across many image pairs.

To make learning tractable and avoid local minima, the model is trained on "infinitesimal" transformations (tiny shifts). Because the generator $G$ represents the intrinsic "physics" of a transformation (e.g., how pixels move during a rotation) independent of the specific object, a $G$ learned from tiny shifts can theoretically be scaled up via the matrix exponential to handle large shifts.

Experimental Validation

The paper relies on synthetically generated datasets to validate the theory, using a periodic, band-limited interpolation method to ensure smooth sub-pixel movement in a discrete pixel space.

  • 1-D Synthetic Images: 20-pixel vectors with uniformly random intensities, subjected to sub-pixel translations.
  • 2-D Synthetic Images: Small grids (from $3\times3$ to $9\times9$) of random pixel intensities, subjected to infinitesimal rotations (0.2 radians).

Key Results

  • 1-D Translation: The algorithm successfully learned the generator for 1-D shifts. The learned matrix $G$ closely matched the analytically derived version. Remarkably, despite being trained on tiny movements, the model could estimate translations as large as 27 pixels with high accuracy.
  • 2-D Rotation: The model learned rotation operators for various spatial scales. For a $5\times5$ image, the learned matrix could rotate a reference image a full 360 degrees.
  • Generative Power: Once $G$ is learned, it doesn't just recognize transformations; it can generate them. A static image can be "rendered" at any rotation or position simply by changing the scalar $x$.

Limitations and Future Directions

While the theoretical foundation is strong, the authors note several limitations. First, while learning $G$ is stable with infinitesimal changes, using the learned $G$ to estimate large transformations ($x$) involves a non-convex optimization landscape prone to local minima. They suggest "coarse-to-fine" strategies to mitigate this. Second, the model struggled with images containing significant high-frequency noise, leading to artifacts during transformation. Finally, future work must test these Lie group operators on realistic natural image sequences, which contain a much richer variety of simultaneous transformations.

Summary updated

Abstract

One of the most important problems in visual perception is that of visual in(cid:173) variance: how are objects perceived to be the same despite undergoing transfor(cid:173) mations such as translations, rotations or scaling? In this paper, we describe a Bayesian method for learning invariances based on Lie group theory. We show that previous approaches based on first-order Taylor series expansions of inputs can be regarded as special cases of the Lie group approach, the latter being ca(cid:173) pable of handling in principle arbitrarily large transfonnations. Using a matrix(cid:173) exponential based generative model of images, we derive an unsupervised al(cid:173) gorithm for learning Lie group operators from input data containing infinites(cid:173) imal transfonnations. The on-line unsupervised learning algorithm maximizes the posterior probability of generating the training data. We provide experimen(cid:173) tal results suggesting that the proposed method can learn Lie group operators for handling reasonably large I-D translations and 2-D rotations.
BibTeX
@inproceedings{rao1998learning,
  author = {Rajesh P. N. Rao and Daniel L. Ruderman},
  title = {{Learning Lie Groups for Invariant Visual Perception}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1998},
  url = {https://openreview.net/forum?id=SkNHePWdZH},
}
Figure