Auto-pooling: Learning to Improve Invariance of Image Features from Image Sequences

Sainbayar Sukhbaatar, Takaki Makino, Kazuyuki AiharaICLR 2013 · arXiv · OpenReview

Introduction

Learning invariant representations from images is one of the hardest challenges in computer vision. Traditionally, "pooling" is the step that makes a model robust to small changes, such as an object shifting slightly to the left. This is typically achieved via spatial pooling (like max-pooling), which groups together features that are physically near each other in an image.

However, spatial pooling has two major drawbacks. First, it is rigid, handling only spatial shifts and relying on manually defined grids (e.g., a 2x2 square). Second, it is restricted to convolutional architectures where features have a clear spatial topology. An ideal pooling mechanism should be learned from data and handle any natural transformation—like rotation or lighting changes—not just simple shifts.

To address this, researchers propose Auto-pooling, a novel unsupervised pooling method that learns to group image features by watching videos rather than relying on hardcoded spatial grids. By optimizing for temporal coherence—the idea that an object’s representation shouldn't change drastically from one video frame to the next—it creates feature representations that are more invariant to complex transformations.

Spatial pooling vs. Auto-pooling

Methods: Pooling by Temporal Coherence

Auto-pooling treats pooling as a "soft clustering" problem. Instead of pooling by location, it pools by temporal coherence. The model is trained on pairs of consecutive frames from videos. The underlying logic is simple: if two frames are just 33 milliseconds apart, they likely contain the same object. Therefore, their pooled representations should be nearly identical.

Auto-pooling training architecture

To achieve this, the authors use a dual-objective cost function:

  • Invariance Cost ($J_1$): Minimizes the distance between the pooled representations of two consecutive frames ($z$ and $z'$). This encourages "slowness" or stability in the features.
  • Reconstruction Cost ($J_2$): Similar to an auto-encoder, this ensures that the original features ($y$) can be reconstructed ($\hat{y}$) from the pooled version. This prevents the model from "cheating" by simply outputting zeros to satisfy the invariance cost.

Unlike spatial pooling, Auto-pooling learns a weight matrix $P$ that defines which features belong to which "cluster." Because it doesn't rely on spatial coordinates, it can be applied to non-convolutional models where the relationship between features isn't strictly geographic.

Key Insights and Generalization

The paper leverages the "slowness principle"—the idea that meaningful properties of the environment change slowly compared to individual pixel values. By forcing the model to be "slow," it naturally learns to be invariant to the transformations occurring in the video.

Auto-pooling discovers clusters of features that traditional pooling would miss. For example, it can group edge detectors with similar orientations but different positions, effectively learning rotation and translation invariance simultaneously. The authors note that this approach mimics how complex cells in the visual cortex might develop, learning from visual sequences during early development rather than being "pre-programmed" with spatial maps.

Furthermore, Auto-pooling shows a significant advantage when training data is scarce. This suggests that the learned invariance helps the model generalize better from fewer examples compared to the "hard" constraints of spatial pooling.

Experimental Results

The researchers evaluated their method using natural videos (44 short videos from Vimeo for initial feature clustering) and the CIFAR-10 dataset (60,000 $32 \times 32$ color images) to test effectiveness in a real-world image classification task.

  • Classification Accuracy: On the CIFAR-10 dataset, Auto-pooling (with 2000 clusters) achieved 69.7% accuracy, surpassing the best traditional spatial pooling configuration (4x4 grid), which reached 68.4%.
  • Small Dataset Performance: The performance gap widened when training on a smaller subset of CIFAR-10 (1,000 samples per class). Auto-pooling reached 65.0%, while spatial pooling lagged behind at 61.5%.
  • Invariance Score: The authors developed a metric to measure how "invariant" a feature is. Auto-pooling significantly increased this score compared to raw features, confirming that the temporal training successfully stabilized the representations.

Limitations and Future Work

While promising, the current experiments primarily focused on low-level features like edge detectors. Applying Auto-pooling within deeper architectures could unlock invariance to even more complex 3D transformations and distortions.

Additionally, training on large amounts of convolutional feature data is computationally intensive. Scaling to much larger datasets or higher-resolution video remains a challenge. Interestingly, Auto-pooling outperformed spatial pooling without even knowing where the features were located. A future hybrid approach that combines learned temporal clusters with basic spatial constraints might yield even better results.

Summary updated

Abstract

Learning invariant representations from images is one of the hardest challenges facing computer vision. Spatial pooling is widely used to create invariance to spatial shifting, but it is restricted to convolutional models. In this paper, we propose a novel pooling method that can learn soft clustering of features from image sequences. It is trained to improve the temporal coherence of features, while keeping the information loss at minimum. Our method does not use spatial information, so it can be used with non-convolutional models too. Experiments on images extracted from natural videos showed that our method can cluster similar features together. When trained by convolutional features, auto-pooling outperformed traditional spatial pooling on an image classification task, even though it does not use the spatial topology of features.
BibTeX
@inproceedings{sukhbaatar2013auto,
  author = {Sainbayar Sukhbaatar and Takaki Makino and Kazuyuki Aihara},
  title = {{Auto-pooling: Learning to Improve Invariance of Image Features from Image Sequences}},
  booktitle = {International Conference on Learning Representations},
  year = {2013},
  url = {https://openreview.net/forum?id=0OR_OycNMzOF9},
}
Figure