Abstract
Zero-Shot Learning (ZSL) aims to recognize unseen object classes by leveraging semantic attributes as a bridge. However, existing methods often struggle with coarse alignment and poor part localization. We present CSAP-ZSL (Class Semantic Attribute Perception Guided Zero-Shot Learning), a dual-layered framework that integrates global contrastive learning with a novel graph-based region partitioning technique. By perceiving attributes at both coarse and fine-grained levels, CSAP-ZSL achieves state-of-the-art performance on standard benchmarks including AWA2, CUB, and SUN, effectively bridging the gap between visual features and abstract semantic descriptions.
Introduction
Zero-Shot Learning is a critical task in computer vision, enabling models to identify objects from classes not seen during training. This is typically achieved by learning a mapping between visual features and a shared semantic attribute space (e.g., "yellow head," "has wings").
Despite its potential, ZSL faces two major challenges: coarse alignment and poor part localization. Many existing methods analyze the image as a whole, missing fine-grained details. Conversely, methods that do focus on parts often use rigid grids or broad attention mechanisms that fail to capture the precise boundaries of semantic attributes.
Methodology
The CSAP-ZSL framework consists of two main modules integrated into a unified network to handle multi-grained perception.
Coarse-grained Perception Module
This module utilizes contrastive semantic learning to align the global embedding of the entire image with the overall class attribute vector. By fine-tuning a Vision Transformer (ViT) backbone, the model learns to extract high-level features that are globally consistent with the target class description.
Fine-grained Perception Module
The core innovation of CSAP-ZSL is its ability to perceive local attributes through two sub-components:
- Soft Region Partitioning (SRP): This component treats image regions as nodes in a graph and uses a graph-cut approach to cluster similar, neighboring regions. This allows the model to group patches into coherent semantic units, such as a bird's head or wing, rather than relying on arbitrary grid boundaries.
- Attribute Prototype Matching Network (APMN): The model learns specific prototypes for each attribute and matches the clustered regions against these prototypes to determine which part of the image best represents a given attribute.
Experimental Results
CSAP-ZSL was evaluated on three standard ZSL benchmarks: Animals with Attributes 2 (AWA2), Caltech-UCSD Birds (CUB), and SUN Attribute (SUN).
Quantitative Performance
The model outperformed 26 state-of-the-art methods across all datasets. The primary metric, the Harmonic Mean (H), showed significant improvements:
- AWA2: Achieved an H-score of 74.6% (surpassing the previous 70.3% benchmark).
- CUB: Achieved an H-score of 70.6%, outperforming TransZero++ and DUET.
- SUN: Reached 52.2%, a notable increase from the previous best of approximately 45%.
Key Insights
- Semantic Grouping: Grouping related regions via graph partitioning allows the model to focus on object parts that correspond directly to human-defined attributes.
- Complementary Learning: The coarse module provides a necessary context that acts as a sanity check for the fine-grained attribute detections.
- Backbone Adaptation: Fine-tuning the ViT backbone is crucial, as off-the-shelf features often lack the nuance required for precise attribute-to-patch alignment.
Conclusion
CSAP-ZSL introduces a robust dual-layered approach to Zero-Shot Learning. By combining global context with sophisticated local region partitioning, it overcomes the limitations of previous coarse-grained and attention-based models. While the graph-based approach adds some computational overhead, the gains in accuracy and interpretability mark a significant step forward for the field.
Summary updated