Locality and compositionality in zero-shot learning

Tristan Sylvain, Linda Petrini, R. Devon Hjelm

Abstract

In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiments show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.

Introduction: Zero-Shot Learning from Scratch

Zero-Shot Learning (ZSL) is the challenging task of recognizing objects from categories that were never seen during the training phase. For instance, a model might be expected to identify a "Zebra" after having seen only "Horses" and being provided with a semantic description like "a horse with stripes."

A significant limitation in current ZSL research is the reliance on feature extractors pre-trained on massive datasets like ImageNet. This "pre-training crutch" makes it difficult to discern whether a model is truly reasoning about novel semantic concepts or simply leveraging visual similarities encountered during pre-training. This paper introduces Zero-Shot Learning from Scratch (ZFS), a rigorous evaluation framework that forbids such pre-training. By training models from the ground up, the authors investigate the fundamental architectural principles—specifically locality and compositionality—that drive generalization.

Methodology: Representation Learning and Metrics

The study evaluates a variety of representation learning methods under the ZFS constraint:

  • Supervised Learning: Standard fully connected (FC) label classifiers.
  • Generative Models: Variational Auto-encoders (VAE, $\beta$-VAE) and Adversarial Auto-encoders (AAE).
  • Self-Supervised Learning: Deep InfoMax (DIM) and Augmented Multiscale DIM (AMDIM).
  • Class-Matching Deep InfoMax (CMDIM): A novel approach introduced in this work that encourages the model to identify common local features across different images of the same class, rather than focusing solely on internal image statistics.

To evaluate these models, the authors employ several key metrics:

  1. Tree Reconstruction Error (TRE): Quantifies compositionality by measuring how well a representation can be reconstructed from a set of primitive semantic attributes.
  2. Parts Classification: Uses a linear probe to determine if learned features accurately capture the location of specific object parts (e.g., beaks or wings).
  3. Mutual Information (MI) Heatmaps: Visualizes which local patches the global representation prioritizes.

Datasets and Evaluation

The research utilizes three standard ZSL benchmarks under the ZFS constraint:

  • CUB (Caltech-UCSD Birds-200-2011): A fine-grained dataset where local features like beaks and wings are critical for identification.
  • AwA2 (Animals with Attributes 2): A dataset of general animal categories with varying object sizes and backgrounds.
  • SUN Attribute: A scene-centric dataset where global context is often more informative than specific local objects.
Examples from the CUB dataset, where fine-grained local parts are essential for zero-shot recognition.
Examples from the AwA2 dataset, illustrating diverse animal categories used to test attribute-based compositionality.

Key Insights and Results

The experiments yielded several critical findings regarding the nature of generalization in ZSL:

The Reconstruction Trap

Models focused on pixel-level reconstruction (like VAEs) performed poorly. These models tend to exhaust their representational capacity on low-level statistics and background details rather than high-level semantic parts necessary for ZSL.

Locality as a Proxy for Generalization

There is a strong correlation (0.73) between a model's ability to localize specific parts (like a bird's beak) and its zero-shot performance. However, "semantic locality" is key; the most successful models, such as CMDIM, learn local features that are shared across a class. CMDIM achieved approximately 35% accuracy on CUB, significantly outperforming standard VAEs (~12%).

The Primacy of Compositionality

Compositionality, measured via the TRE metric, emerged as the strongest predictor of ZSL success. A massive negative correlation (-0.90 on CUB) was found between TRE and ZSL error, indicating that models which naturally organize internal logic into a "vocabulary" of parts and attributes generalize far better to unseen classes.

Conclusion

This work demonstrates that locality and compositionality are not just desirable traits but fundamental drivers of generalization in Zero-Shot Learning. By removing the influence of pre-training, the ZFS framework highlights the need for models that prioritize semantic parts over global pixel reconstruction. Future research should focus on scaling these principles to larger architectures and developing methods to automatically discover object parts without manual annotations.

Summary updated

Abstract

In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiments show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.
BibTeX
@inproceedings{sylvain2020locality,
  author = {Tristan Sylvain and Linda Petrini and R. Devon Hjelm},
  title = {{Locality and compositionality in zero-shot learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=6CEKwWjuVQ},
}
Figure