Abstract
Most existing Zero-Shot Learning (ZSL) approaches learn direct embeddings from global features or image parts (regions) to the semantic space, which fail to capture the appearance relationships between different local regions within a single image. In this paper, we incorporate region-based relation reasoning into ZSL via the Region Graph Embedding Network (RGEN). RGEN consists of two branches: the Constrained Part Attention (CPA) branch, which produces image regions, and the Parts Relation Reasoning (PRR) branch, which models interactions among these regions using graph convolutions. We introduce transfer and balance losses to contrast class similarities and pursue maximum response consistency. Extensive experiments validate RGEN's effectiveness under both ZSL and generalized ZSL (GZSL) settings.
Introduction: The Challenge of Context and Bias
Zero-Shot Learning (ZSL) aims to recognize objects from "unseen" categories by leveraging semantic descriptions (attributes) shared with "seen" categories. For instance, a model trained on horses and tigers might recognize a zebra by understanding it has "stripes" and "four legs."
However, traditional ZSL methods face two major hurdles:
- Lack of Relational Context: Many models rely on global image features or isolated regions. They miss the crucial spatial and appearance relationships—like how a head relates to a torso—that define an object's identity.
- Seen-Class Bias in GZSL: In Generalized ZSL (GZSL), where test sets include both seen and unseen classes, models often exhibit a strong bias toward seen classes, leading to high misclassification rates for unseen objects.
The RGEN Architecture
The proposed Region Graph Embedding Network (RGEN) is an end-to-end framework designed to capture part-to-part relationships and mitigate domain bias.
Constrained Part Attention (CPA)
The CPA branch employs soft spatial attention to identify $K$ discriminative regions. To ensure these regions are distinct and informative, the authors apply compactness and divergence losses. These constraints force the attention masks to focus on specific, non-overlapping parts of the object, providing a rich set of local features.
Parts Relation Reasoning (PRR)
The PRR branch is the core innovation of RGEN. It represents the discovered regions as nodes in a Region Graph, where edges denote appearance similarity. A two-layer Graph Convolutional Network (GCN) performs reasoning over this graph, allowing the model to learn higher-order relational features. This ensures the model recognizes that a specific configuration of parts (e.g., "striped body" near "horse-like head") is characteristic of a zebra.
Optimized Training Losses
RGEN utilizes two specialized losses to improve performance:
- Transfer Loss: Enhances knowledge transfer by using Least Square Regression (LSR) to map similarities between seen and unseen class attributes.
- Balance Loss: Addresses the domain bias in GZSL by encouraging the prediction scores for seen and unseen classes to be on a similar numerical scale. This prevents the model from being overconfident in seen classes at the expense of unseen ones.
Experimental Results and Insights
RGEN was evaluated on four benchmark datasets: CUB (birds), AWA2 (animals), SUN (scenes), and APY (diverse objects).
Performance Benchmarks
RGEN achieved state-of-the-art (SOTA) results across multiple settings:
- ZSL Accuracy: Reached 76.1% on CUB and 73.6% on AWA2, outperforming previous end-to-end models.
- GZSL Performance: On AWA2, RGEN achieved a Harmonic Mean (H-score) of 71.5%, a significant improvement over previous methods.
Key Takeaways
- Relational Reasoning is a Game Changer: The PRR branch significantly boosts performance, proving that spatial and appearance context is vital for semantic transfer.
- Balancing the Scale: The Balance Loss effectively reduces the misclassification of unseen objects into seen categories by normalizing response scales.
- End-to-End Advantage: Learning part-based features end-to-end is superior to using static, pre-extracted global features from a backbone like ResNet.
Conclusion
RGEN demonstrates that modeling the relationships between local image regions is a powerful strategy for Zero-Shot Learning. By combining graph-based reasoning with targeted losses to combat seen-class bias, RGEN sets a new standard for recognizing novel objects in complex environments. While challenges remain in data-scarce scenarios like the SUN dataset, RGEN provides a robust framework for bridging the gap between visual parts and semantic attributes.
Summary updated