Locality and Compositionality in Zero-Shot Learning

Tristan Sylvain, Linda Petrini, R. Devon Hjelm

Introduction

Zero-Shot Learning (ZSL) presents a formidable challenge in computer vision: a model must recognize a new category (e.g., a "Zebra") having only seen other animals (e.g., "Horses" and "Tigers") alongside a text description of the new category's attributes (e.g., "has stripes").

Many state-of-the-art ZSL models rely on features pre-trained on massive datasets like ImageNet. While this yields high accuracy, it obscures the underlying principles of representation learning. It becomes difficult to determine if the model is actually "reasoning" about attributes or just benefiting from having seen similar pixels during pre-training. To isolate these variables, researchers propose Zero-Shot Learning from Scratch (ZFS). By forbidding massive pre-training, this benchmark asks a fundamental question: What specific properties of a representation allow it to generalize to unseen concepts?

This study demonstrates that locality (focusing on small, specific parts of an image) and compositionality (building complex concepts from simpler building blocks) are the primary drivers of a model's ability to recognize novel objects.

Methods: Learning from Scratch

The authors compare a wide variety of representation learning strategies under the strict ZFS constraint:

  • Supervised: Standard label classifiers.
  • Generative/Reconstruction: Variational Autoencoders (VAEs), $\beta$-VAEs, and Adversarial Auto-encoders (AAEs).
  • Self-Supervised: Augmented Multiscale Deep InfoMax (AMDIM) and a newly introduced variant called Class-Matching DIM (CMDIM). CMDIM encourages the model to find common features between different images of the same class, pushing it toward more discriminative, semantic representations.
  • Auxiliary Local Losses: The introduction of local attribute-based (AC) and label-based (LC) losses forces the model to learn semantically relevant features at early stages of the network.

To quantitatively measure their hypotheses, the researchers used the Tree Reconstruction Error (TRE) as a proxy for compositionality and a Parts F1 score (using bird part annotations) to measure locality.

Datasets

The study utilizes three standard ZSL benchmarks, each representing different visual challenges:

  1. CUB (Caltech-UCSD Birds): Fine-grained classification where small parts (beaks, wings) are crucial.
  2. AwA2 (Animals with Attributes 2): Coarse-grained classification of animals of varying sizes.
  3. SUN Attribute: A scene-based dataset where the "object" is often the entire image.
Typical samples from CUB, AwA2, and SUN datasets illustrating varying expressions of compositionality and locality.

Key Insights

By evaluating these models without the crutch of ImageNet, several clear patterns emerged regarding what makes a representation generalize:

  • Reconstruction is a Trap for ZSL: Generative models like VAEs performed poorly. Because they are forced to reconstruct every pixel (including the background), they waste their "representational budget" on non-semantic information. They become too sensitive to pixel-level statistics and fail to capture the high-level, reusable "parts" needed for ZSL.
  • Locality is Predictive: There is a strong correlation (0.73) between a model's ability to correctly identify local parts (like a bird's tail) and its overall ZSL accuracy. Models that "know" where the parts are generalize better.
  • Compositionality is the Secret Sauce: The authors found a massive correlation (up to -0.90 on the CUB dataset) between the TRE ratio (compositionality) and ZSL performance. This suggests that the more a model treats an image as a collection of reusable "primitives," the better it can describe a brand-new category.
  • Semantic Locality Matters: Simply having a small receptive field isn't enough. The local features must share information with the global concept (as seen in CMDIM) to be genuinely useful for generalization.

Key Results

The proposed Class-Matching DIM (CMDIM) outperformed other unsupervised and generative methods, showing that matching features across different instances of a class is a powerful way to learn ZSL-ready representations.

Furthermore, adding an auxiliary local attribute loss (AC) improved performance across almost all models. For example, on the CUB dataset, adding local losses significantly boosted both the "Parts F1" score and overall ZSL accuracy simultaneously. The study confirmed that as representations become more compositional (indicated by a lower TRE ratio), ZSL accuracy increases linearly across all three datasets.

Limitations and Future Work

The primary limitation of this study is the scale of the encoders. To keep the "from scratch" training computationally feasible, the authors used smaller architectures (modified DCGAN and AlexNet) rather than the massive ResNet-101 backbones common in standard ZSL literature. Consequently, the absolute accuracy scores are lower than state-of-the-art models that utilize ImageNet pre-training.

Future research could explore how to scale these "locality and compositionality" constraints to larger architectures and more complex datasets without losing training stability. Ultimately, this work suggests that future representation learning should move away from pure pixel reconstruction and toward more "local-aware" objectives.

Summary updated

Abstract

In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiments show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.
BibTeX
@inproceedings{sylvain2020locality,
  author = {Tristan Sylvain and Linda Petrini and R. Devon Hjelm},
  title = {{Locality and Compositionality in Zero-Shot Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=6CEKwWjuVQ},
}
Figure