Simple and effective localized attribute representations for zero-shot learning

Shiqi Yang, Kai Wang, Luis Herranz, Joost van de Weijer

TL;DR

This paper introduces SELAR, a novel approach to zero-shot learning (ZSL) that localizes attribute representations in the semantic space, rather than the visual space. This implicit localization, achieved by reordering projection and aggregation steps, leads to state-of-the-art performance on fine-grained datasets like CUB and SUN, outperforming more complex methods.

What Problem Does This Paper Address?

Deep convolutional neural networks have revolutionized image classification, but they typically require large amounts of labeled data per class. This is a significant limitation in real-world scenarios where data for new classes might be scarce or non-existent. Zero-shot learning (ZSL) aims to bridge this gap by enabling models to recognize unseen classes based on their semantic descriptions (e.g., attributes, word embeddings).

Traditional ZSL methods often rely on global visual features, which can struggle with fine-grained distinctions where subtle local details are crucial for discrimination (e.g., the color of a bird's beak). While some methods attempt to address this by explicitly localizing features in the visual space using attention or part detection modules, these approaches are often complex, require additional annotations, and can be difficult to train. This paper addresses the need for a simpler, yet effective, method to leverage localized information for ZSL, particularly in fine-grained recognition tasks.

Methods

The core contribution of this paper is the Simple and Effective Localized Attribute Representation (SELAR). Unlike previous methods that localize in the visual space, SELAR localizes in the semantic/attribute space. This is achieved by reordering the standard ZSL pipeline: instead of extracting visual features, aggregating them, and then projecting to the semantic space, SELAR projects local visual features to the semantic space first, and then aggregates these localized semantic representations.

Key aspects of the SELAR method include:

  • Implicit Localization: SELAR doesn't require explicit attention or part detection modules. Instead, it leverages the implicit localization capabilities of the feature extractor and reorders the processing pipeline.
  • Localized Attributes: Each feature map in the localized attribute space corresponds to a specific attribute, leading to disentangled and interpretable representations.
  • Spatial Aggregation: The paper investigates two spatial aggregation strategies: Global Average Pooling (GAP) and Global Maximum Pooling (GMP). They find that GMP, when applied in the attribute space, is particularly effective, especially for fine-grained datasets.
  • Simple Pipeline: The method is designed to be simple, using a standard cross-entropy loss and avoiding additional hyperparameters, making it easy to implement and train.

Datasets

The authors evaluate their method on three standard ZSL datasets:

  • CUB (Caltech-UCSD Birds 200): A fine-grained dataset of bird images with 200 classes and 312 attributes.
  • SUN (Scene UNderstanding): A dataset of scene images with 717 classes and 102 attributes.
  • AWA2 (Animals with Attributes 2): A dataset of animal images with 50 classes and 85 attributes.

The paper uses the proposed split from Xian et al. (2018) for evaluation, which is common in ZSL/GZSL research.

Key Insights and Ideas

  • Semantic Space Localization is Key: The paper argues that localizing in the semantic space is more effective for ZSL than localizing in the visual space, especially for fine-grained tasks. This allows for disentangled attribute representations.
  • Order Matters for Aggregation: The order of projection and aggregation significantly impacts performance, particularly when using non-linear aggregation methods like GMP. Aggregating in the semantic space after projection is crucial.
  • GMP Outperforms GAP for Fine-Grained Tasks: For datasets with highly localized attributes (like CUB), GMP is shown to be more effective than GAP, generating sparser and more discriminative feature patterns.
  • Simplicity Leads to Robustness: The proposed SELAR method's simplicity, with fewer hyperparameters, contributes to its strong generalization and reduced bias towards seen classes, which is critical for Generalized Zero-Shot Learning (GZSL).

Key Results

  • State-of-the-Art Performance: SELAR, particularly with GMP and fine-tuned feature extractors, achieves state-of-the-art results on the CUB and SUN datasets for GZSL.
  • Competitive Results on AWA2: The method also achieves competitive results on the AWA2 dataset, outperforming more complex methods.
  • Reduced Bias: SELAR demonstrates a lower bias towards seen classes compared to many other methods, as evidenced by the S/U ratio in GZSL.
  • Interpretable Attribute Maps: The method generates attribute-specific heatmaps, providing visual interpretability of which regions are associated with specific attributes.

Limitations and Future Work

  • Attribute Map Guarantee: The authors acknowledge that they cannot guarantee that each attribute map perfectly corresponds to the true attribute without additional constraints. While high attribute values in the vector correlate with relevant regions, this correspondence is learned implicitly.
  • Dataset Specificity: While SELAR performs well on fine-grained datasets, the paper notes that for datasets like SUN, where attributes are less localized, the performance difference between GAP and GMP is less pronounced.
  • Future Work: The paper suggests that their simple pipeline can serve as a new baseline for zero-shot learning. Further exploration into attribute map interpretation and potential combinations with generative methods for data augmentation are implied directions.

Summary updated

Abstract

Zero-shot learning (ZSL) aims to discriminate images from unseen classes by exploiting relations to seen classes via their semantic descriptions. Some recent papers have shown the importance of localized features together with fine-tuning the feature extractor to obtain discriminative and transferable features. However, these methods require complex attention or part detection modules to perform explicit localization in the visual space. In contrast, in this paper we propose localizing representations in the semantic/attribute space, with a simple but effective pipeline where localization is implicit. Focusing on attribute representations, we show that our method obtains state-of-the-art performance on CUB and SUN datasets, and also achieves competitive results on AWA2 dataset, outperforming generally more complex methods with explicit localization in the visual space. Our method can be implemented easily, which can be used as a new baseline for zero shot-learning. In addition, our localized representations are highly interpretable as attribute-specific heatmaps.
BibTeX
@misc{yang2020simple,
  author = {Shiqi Yang and Kai Wang and Luis Herranz and Joost van de Weijer},
  title = {{Simple and effective localized attribute representations for zero-shot learning}},
  year = {2020},
  eprint = {2006.05938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.05938},
}

Loading related content...

Figure