Class Semantic Attribute Perception Guided Zero-Shot Learning

Qin Yue, Junbiao Cui, Jianqing Liang, Liang Bai

Abstract

Zero-Shot Learning (ZSL) aims to recognize unseen object classes by leveraging semantic attributes as a bridge. However, existing methods often struggle with coarse alignment and poor part localization. We present CSAP-ZSL (Class Semantic Attribute Perception Guided Zero-Shot Learning), a dual-layered framework that integrates global contrastive learning with a novel graph-based region partitioning technique. By perceiving attributes at both coarse and fine-grained levels, CSAP-ZSL achieves state-of-the-art performance on standard benchmarks including AWA2, CUB, and SUN, effectively bridging the gap between visual features and abstract semantic descriptions.

Introduction

Zero-Shot Learning is a critical task in computer vision, enabling models to identify objects from classes not seen during training. This is typically achieved by learning a mapping between visual features and a shared semantic attribute space (e.g., "yellow head," "has wings").

Despite its potential, ZSL faces two major challenges: coarse alignment and poor part localization. Many existing methods analyze the image as a whole, missing fine-grained details. Conversely, methods that do focus on parts often use rigid grids or broad attention mechanisms that fail to capture the precise boundaries of semantic attributes.

Comparison of ZSL paradigms: (a) global alignment, (b) attention-based part localization, and (c) the proposed region-partitioning approach.

Methodology

The CSAP-ZSL framework consists of two main modules integrated into a unified network to handle multi-grained perception.

Coarse-grained Perception Module

This module utilizes contrastive semantic learning to align the global embedding of the entire image with the overall class attribute vector. By fine-tuning a Vision Transformer (ViT) backbone, the model learns to extract high-level features that are globally consistent with the target class description.

Fine-grained Perception Module

The core innovation of CSAP-ZSL is its ability to perceive local attributes through two sub-components:

  • Soft Region Partitioning (SRP): This component treats image regions as nodes in a graph and uses a graph-cut approach to cluster similar, neighboring regions. This allows the model to group patches into coherent semantic units, such as a bird's head or wing, rather than relying on arbitrary grid boundaries.
  • Attribute Prototype Matching Network (APMN): The model learns specific prototypes for each attribute and matches the clustered regions against these prototypes to determine which part of the image best represents a given attribute.
The CSAP-ZSL architecture featuring Coarse-grained and Fine-grained Perception Modules.

Experimental Results

CSAP-ZSL was evaluated on three standard ZSL benchmarks: Animals with Attributes 2 (AWA2), Caltech-UCSD Birds (CUB), and SUN Attribute (SUN).

Quantitative Performance

The model outperformed 26 state-of-the-art methods across all datasets. The primary metric, the Harmonic Mean (H), showed significant improvements:

  • AWA2: Achieved an H-score of 74.6% (surpassing the previous 70.3% benchmark).
  • CUB: Achieved an H-score of 70.6%, outperforming TransZero++ and DUET.
  • SUN: Reached 52.2%, a notable increase from the previous best of approximately 45%.

Key Insights

  • Semantic Grouping: Grouping related regions via graph partitioning allows the model to focus on object parts that correspond directly to human-defined attributes.
  • Complementary Learning: The coarse module provides a necessary context that acts as a sanity check for the fine-grained attribute detections.
  • Backbone Adaptation: Fine-tuning the ViT backbone is crucial, as off-the-shelf features often lack the nuance required for precise attribute-to-patch alignment.

Conclusion

CSAP-ZSL introduces a robust dual-layered approach to Zero-Shot Learning. By combining global context with sophisticated local region partitioning, it overcomes the limitations of previous coarse-grained and attention-based models. While the graph-based approach adds some computational overhead, the gains in accuracy and interpretability mark a significant step forward for the field.

Summary updated

Abstract

Deep learning has achieved remarkable success in supervised image classification tasks, which relies on a large number of labeled samples for each class. Recently, zero-shot learning has garnered significant attention, which aims to recognize unseen classes using only training samples from seen classes. To bridge the gap between images and classes, class semantic attributes are introduced, making the alignment between image and class semantic attributes critical to zero-shot learning. However, existing methods often struggle to accurately focus on the image regions corresponding to individual class semantic attributes and tend to overlook the relations between different regions of an image, leading to poor alignment. To address these challenges, we propose a class semantic attribute perception guided zero-shot learning method. Specifically, we achieve coarse-grained perception of class semantic attributes across the entire image through contrastive semantic learning. Additionally, we attain fine-grained perception of individual class semantic attributes within image regions via region partitioning-based attribute alignment, which fully considers the relations between different regions of an image. By integrating these two processes into a unified network, we achieve multi-grained class semantic attribute perception, thereby enhancing the alignment between images and class semantic attributes. We validate the effectiveness of the proposed method on zero-shot learning benchmark data sets.
BibTeX
@inproceedings{yue2025class,
  author = {Qin Yue and Junbiao Cui and Jianqing Liang and Liang Bai},
  title = {{Class Semantic Attribute Perception Guided Zero-Shot Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34383},
}
Figure