Dual Feature Augmentation Network for Generalized Zero-shot Learning

Lei Xiang, Yuan Zhou, Haoran Duan, Yang Long

Generalized Zero-Shot Learning (GZSL) is a critical subfield of computer vision where models are tasked with recognizing objects from classes they have never encountered during training. This requires a robust bridge between visual features (pixels) and semantic attributes (e.g., "has a yellow beak"). However, standard approaches often struggle with three key issues: feature entanglement (overlapping visual features for different attributes), attribute diversity (the mismatch between static class-level attributes and image-specific variations), and granularity conflicts (the difficulty of using a single function to map both global and local features).

The Dual Feature Augmentation Network (DFAN)

To address these challenges, the authors propose the Dual Feature Augmentation Network (DFAN). The architecture, illustrated in Figure 1, is built around two primary augmentation modules designed to refine the learning process:

  • Visual Feature Augmentation: This module uses an attention mechanism to pinpoint specific attributes within an image. To combat entanglement, a Cosine Similarity Loss ($L_{cos}$) is employed to ensure that features for different attributes are orthogonal in the embedding space, effectively decoupling them.
  • Semantic Feature Augmentation (Bias Learner): Recognizing that individual images vary from the "average" class attribute, a 3-layer Multi-Layer Perceptron (MLP) called a Bias Learner predicts an image-specific offset. This adjusts the static attributes to better match the visual evidence in a specific sample.
  • Dual Predictors: The model utilizes separate Local and Global Predictors to handle different levels of visual granularity, avoiding the optimization conflicts inherent in single-mapping approaches.
The DFAN architecture featuring Visual and Semantic Feature Augmentation modules, including the Bias Learner and dual predictors.

Experimental Evaluation

The DFAN was evaluated on three standard benchmark datasets: CUB (fine-grained birds), SUN (scenes), and AWA2 (coarse-grained animals).

As shown in Figure 2, DFAN achieves state-of-the-art performance, particularly on fine-grained recognition. On the CUB dataset, it reached a Harmonic mean (H) of 71.8% in the GZSL setting, outperforming prominent non-generative methods like TransZero and MSDN. On the SUN dataset, it achieved the highest Top-1 accuracy (67.9%) in the Conventional ZSL (CZSL) setting.

Comparison of DFAN with state-of-the-art generative and non-generative ZSL methods across three benchmark datasets.

Ablation studies further confirmed the design choices. For instance, the transition from a single predictor to dual predictors resulted in a significant performance boost (up to 11% on CUB), validating the necessity of separating local and global feature processing to resolve granularity conflicts.

Conclusion and Future Work

DFAN demonstrates that decoupling visual features and allowing for image-specific semantic flexibility significantly improves zero-shot recognition. While highly effective for fine-grained tasks where local parts are distinct, the model's performance on coarse-grained datasets like AWA2 suggests that future work could focus on better integrating holistic, multi-scale attention to handle attributes that describe global content rather than specific parts.

Summary updated

Abstract

Abstract
BibTeX
@inproceedings{xiang2023dual,
  author = {Lei Xiang and Yuan Zhou and Haoran Duan and Yang Long},
  title = {{Dual Feature Augmentation Network for Generalized Zero-shot Learning}},
  booktitle = {BMVC},
  year = {2023},
  url = {https://openreview.net/forum?id=m5QBDxIadY},
}
Figure