"Mental Rotation" by Optimizing Transforming Distance

Weiguang Ding, Graham W. TaylorarXiv 2014 · OpenReview

TL;DR

This paper introduces a novel "transforming distance" metric inspired by human "mental rotation." It uses a trained relational model to actively transform image representations, allowing for more robust object recognition by optimizing similarity under learned transformation constraints, rather than relying solely on invariant features.

What Problem Does This Paper Address?

Object recognition systems often struggle with variations in an object's appearance due to transformations like changes in viewpoint, lighting, or expression. While many approaches aim to achieve invariance to these transformations (i.e., the representation stays the same regardless of the transformation), this paper argues that humans don't just ignore transformations; they actively simulate them. This "mental rotation" process allows humans to compare objects even when they are presented differently. The paper addresses the gap of explicitly modeling this mental simulation process within machine learning for improved recognition, especially in scenarios where purely invariant representations might lose crucial information or where databases are sparse.

Methods

The core of the proposed method is a "transforming distance" that leverages a factored gated restricted Boltzmann machine (fgRBM).

  • Relational Model (fgRBM): The fgRBM is trained on pairs of images (e.g., different expressions of the same face, or the same object from different views). It learns a latent representation, h, that encodes the transformation between these image pairs. This h can then be used to represent a "valid" transformation.
  • Transforming Function t(x, h): Using the learned h, a function t(x, h) is defined to transform an input image x into a new representation that respects the learned transformation constraints. This is essentially a learned "mental rotation" of the image's representation.
  • Transforming Distance: The paper proposes two types of transforming distances:
  • Single-sided: Transforms one image (x^a) to match another (x^b).
  • Dual-sided: Transforms both images (x^a and x^b) to match each other.
  • Optimization: To compute the transforming distance between two images, an optimization process is used. It aims to find the optimal latent representation h* that minimizes a cost function. This cost function balances two terms:
  • The distance between the transformed image and the target image (using a standard pairwise distance like Euclidean).
  • A regularization term based on the fgRBM's free energy, ensuring that the transformation h* is "valid" according to what the model learned.
  • Transforming KNN: The proposed transforming distance is then used to replace the standard distance metric in a K-Nearest Neighbors (KNN) classifier.

Datasets

The paper evaluates its method on two datasets:

  • Toronto Face Database (TFD): Contains face images. The focus is on learning expression transformations. The dataset is split into several sets for training, validation, and testing, with specific configurations (TFDs1, TFDs2) to test different scenarios, including imbalanced identities.
  • NORB Dataset: Contains stereo image pairs of toys. The focus is on learning rotation (azimuth) transformations. The "small NORB" variant is used, with a clean background.

Key Insights and Ideas

  • Beyond Invariance: The paper challenges the sole focus on invariant representations, suggesting that actively modeling transformations ("mental rotation") can be more powerful.
  • Learned Transformations as Constraints: The fgRBM doesn't just learn to map one image to another; it learns the space of valid transformations, which is then used to guide the similarity comparison.
  • Regularized Optimization: The key to making the transforming distance effective is the regularization term, which prevents the optimization from producing arbitrary transformations that merely minimize pixel-level distance but violate learned structural constraints.
  • Data Augmentation Potential: The relational model can be used to generate synthetic training data, effectively augmenting the database and improving performance, especially in sparse data scenarios.

Key Results

  • Improved KNN Accuracy: Transforming KNN consistently outperforms regular KNN and an "augmented KNN" on both TFD and NORB datasets.
  • On TFD, gains of 6-10% (TFDs1) and up to 25% (TFDs2) are reported.
  • On NORB, a 4% improvement is seen with the "SmallTrans" model.
  • Robustness to Missing Data: Transforming KNN shows remarkable robustness when the KNN database is significantly reduced (up to 99% missing data), maintaining performance where regular KNN degrades severely. This highlights the "volume coverage" provided by the transforming distance.
  • Qualitative Evidence: Visualizations show that the learned transformations preserve object identity (e.g., faces remain recognizable) even when transforming between different identities, and rotations are preserved for objects.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that performing inference and optimization over latent variables is computationally more demanding than traditional similarity measures. While GPU parallelization offers some relief, it's a significant hurdle for large-scale problems.
  • Approximate Methods: The method precludes the use of approximate nearest neighbor techniques, which are common for large databases.
  • Single Transformation Class: In the experiments, only one type of transformation (expression or rotation) was learned at a time. The authors suggest that future work should explore learning a richer set of transformations using relational models like fgRBMs.
  • Generalization: Demonstrating the efficacy of transforming similarity across a wider range of transformations is a key future direction.

Summary updated

Abstract

The human visual system is able to recognize objects despite transformations that can drastically alter their appearance. To this end, much effort has been devoted to the invariance properties of recognition systems. Invariance can be engineered (e.g. convolutional nets), or learned from data explicitly (e.g. temporal coherence) or implicitly (e.g. by data augmentation). One idea that has not, to date, been explored is the integration of latent variables which permit a search over a learned space of transformations. Motivated by evidence that people mentally simulate transformations in space while comparing examples, so-called "mental rotation", we propose a transforming distance. Here, a trained relational model actively transforms pairs of examples so that they are maximally similar in some feature space yet respect the learned transformational constraints. We apply our method to nearest-neighbour problems on the Toronto Face Database and NORB.
BibTeX
@misc{ding2014mental,
  author = {Weiguang Ding and Graham W. Taylor},
  title = {{"Mental Rotation" by Optimizing Transforming Distance}},
  year = {2014},
  eprint = {1406.3010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1406.3010},
}

Loading related content...

Figure