Representation Learning Through Latent Canonicalizations

Or Litany, Ari Morcos, Srinath Sridhar, Leonidas J. Guibas, Judy HoffmanOpenReview 2020 · DOI · OpenReview

Abstract

We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision. Many prior approaches to this problem have focused on learning "disentangled" representations so that as individual factors vary in a new domain, only a portion of the representation need be updated. In this work, we seek the generalization power of disentangled representations, but relax the requirement of explicit latent disentanglement and instead encourage linearity of individual factors of variation by requiring them to be manipulable by learned linear transformations. We dub these transformations latent canonicalizers, as they aim to modify the value of a factor to a pre-determined (but arbitrary) canonical value (e.g., recoloring the image foreground to black). Assuming a source domain with access to meta-labels specifying the factors of variation within an image, we demonstrate experimentally that our method helps reduce the number of observations needed to generalize to a similar target domain when compared to a number of supervised baselines.

Introduction

Deep learning models are notoriously data-hungry, often requiring massive labeled datasets to generalize. A common strategy to bypass this is Sim2Real—training on synthetic data from simulators where labels are "free" and then fine-tuning on real-world data. However, simulators often have a "domain gap"; they might capture the core task but vary wildly in "nuisance" factors like lighting, background textures, or camera angles.

Previous research attempted to solve this via disentanglement, trying to force the model to store specific attributes in separate latent dimensions. This is mathematically difficult and often unnecessary for downstream tasks. We propose latent canonicalization: instead of caring where information is stored, we ensure it can be predictably manipulated via learned linear transformations to map specific attributes to fixed "canonical" values.

Methodology

The framework combines a Denoising Autoencoder (AE) with learned Latent Canonicalizers.

The latent canonicalization architecture. An encoder maps input $x$ to latent $z$. Canonicalization matrices $C_j$ map $z$ to $z_{canon}$, which the decoder reconstructs into canonical images. The total loss $\mathcal{L}$ combines classification, bypass (reconstruction), and canonicalization terms.

Architecture and Latent Canonicalizers

An encoder maps an image $x$ to a latent vector $z$. For every known factor of variation in the synthetic data (e.g., font size, background color), the model learns a unique linear transformation matrix $C_j$. When $z$ is multiplied by $C_j$, the resulting vector $z_{canon}$ represents the original image but with that specific factor "clamped" to a fixed, arbitrary value (e.g., "all foregrounds are now black").

Training Objective

The model is trained using a multi-task objective:

  • Reconstruction Loss: Standard autoencoder behavior to preserve image information.
  • Canonicalization Loss: Comparing the image generated from $z \cdot C_j$ to a ground-truth "canonical" image provided by the simulator.
  • Classification Loss: A supervised task (like digit recognition) performed on the latent $z$.

Because the transformations are linear, they are compositional; multiple factors can be neutralized simultaneously by chaining matrices (e.g., $z \cdot C_{color} \cdot C_{rotation}$).

Experimental Evaluation

We evaluated the method across three tiers of complexity:

  • dSprites: A toy dataset of 2D shapes used to validate factor isolation (scale, rotation, position).
  • SVHN (Street View House Numbers): The primary benchmark for Sim2Real. We used a custom simulator to generate synthetic digits with known meta-labels (font, color, shear, etc.) and evaluated on the real SVHN test set.
  • SynthImageNet: Naturalistic images created using 3D models from ShapeNet rendered with various textures and camera angles.

Results and Discussion

Qualitative results on the SVHN dataset. The model successfully canonicalizes individual factors (e.g., font size, rotation) and combinations thereof, producing accurate image predictions $\hat{x}_{canon}$ that match targets $x_{canon}$.

Sample Efficiency and Sim2Real Performance

On the SVHN task, the model achieved 5x better sample efficiency than standard baselines. Its performance with only 20 labeled real-world examples matched a standard AE + Classifier using 100 examples. In low-shot settings (10–100 samples), the proposed method outperformed all baselines, including self-supervised methods like RotNet. For 10-shot SVHN, our method with majority voting achieved 83.41% accuracy vs. the baseline's 79.61%.

Latent Augmentation

Since the model can "neutralize" nuisance factors in the latent space, it can perform latent augmentation at test time. By applying different canonicalizers to create multiple "versions" of a real image's latent vector and having the classifier vote on the result, accuracy is consistently boosted.

Conclusion

Latent canonicalization creates a structured, linear latent space that allows models to generalize from synthetic to real-world data with high efficiency. By forcing factors of variation to be linearly manipulable, the model learns a representation that is easier for simple classifiers to navigate, even when visual styles change between simulation and reality.

Summary updated

Abstract

We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision. Many prior approaches to this problem have focused on learning "disentangled" representations so that as individual factors vary in a new domain, only a portion of the representation need be updated. In this work, we seek the generalization power of disentangled representations, but relax the requirement of explicit latent disentanglement and instead encourage linearity of individual factors of variation by requiring them to be manipulable by learned linear transformations. We dub these transformations latent canonicalizers, as they aim to modify the value of a factor to a pre-determined (but arbitrary) canonical value (e.g., recoloring the image foreground to black). By creating simple simulators with pre-specified factors of variation to roughly approximate datasets such as SVHN and ImageNet, we demonstrate experimentally that our method helps reduce the number of observations needed to generalize to a similar target domain when compared to a number of supervised baselines.
BibTeX
@misc{litany2020representation,
  author = {Or Litany and Ari Morcos and Srinath Sridhar and Leonidas J. Guibas and Judy Hoffman},
  title = {{Representation Learning Through Latent Canonicalizations}},
  year = {2020},
  eprint = {2002.11829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.11829},
  howpublished = {OpenReview},
}
Figure