Zero-Shot Learning from Adversarial Feature Residual to Compact Visual Feature

Bo Liu, Qiulei Dong, Zhanyi HuAAAI 2020 · DOI

Abstract

Zero-Shot Learning (ZSL) faces significant challenges due to distribution overlap and semantic-visual inconsistency in synthesized features. This paper introduces AFRNet (Adversarial Feature Residual Network), a framework that shifts the generative focus from full visual features to adversarial "residuals." By anchoring these residuals to predicted class prototypes and employing a novel feature selection strategy, AFRNet produces compact, well-separated visual features that significantly improve recognition accuracy for unseen classes.

Introduction

Zero-Shot Learning (ZSL) aims to recognize objects from classes that were never seen during training by leveraging semantic descriptions (e.g., attributes or text). While Generative Adversarial Networks (GANs) have become a standard tool for synthesizing visual features for these unseen classes, they often suffer from two major issues: distribution overlap, where synthetic features for different classes bleed into each other, and semantic-visual inconsistency, where generic attributes fail to capture class-specific visual nuances. AFRNet addresses these by generating instance-specific residuals relative to a stable class prototype.

The AFRNet Framework

The core innovation of AFRNet is the transition from generating entire feature vectors to generating the numerical difference—the residual—between a specific instance and its class prototype.

Comparison of standard feature generation (A) versus the proposed residual-based approach (B).

As illustrated in Figure 1, traditional methods (A) attempt to map semantic features directly to a cloud of visual features. In contrast, AFRNet (B) uses a Prototype Predictor to establish a central anchor for the class and a Residual Generator to model the local variance. Because these residuals are numerically small relative to the distances between class prototypes, the resulting synthetic clusters remain distinct and compact.

Detailed Architecture and Feature Selection

The AFRNet pipeline consists of several integrated components designed to ensure the reliability of the generated features:

  1. Prototype Predictor: Uses Support Vector Regression (SVR) to predict a visual prototype (mean feature vector) from semantic attributes.
  2. Feature Selection Strategy (FSS): Not all visual dimensions are equally predictable from text. FSS adaptively selects the $K$ dimensions with the lowest prediction error, creating a "compact" prototype that filters out noise.
  3. Residual Generator: A GAN-based component that generates the instance-specific residual.
  4. Synthesis: The final feature is the sum of the compact prototype and the generated residual.
The complete AFRNet architecture, detailing the Feature Generation Phase and the subsequent Classification Phase.

Experimental Results

AFRNet was evaluated across six major ZSL benchmarks, including fine-grained (CUB, NAB) and coarse-grained (APY, AWA1, AWA2, SUN) datasets. The model achieved state-of-the-art performance, with particularly notable gains in challenging scenarios:

  • Conventional ZSL: On the APY dataset, AFRNet reached 56.2% accuracy, a 12.2% improvement over previous benchmarks.
  • Fine-Grained ZSL: On the CUB dataset (SCS-split), it achieved 50.3%, outperforming the previous best method (GAZSL) by 6.6%.
  • Generalized ZSL (GZSL): The model demonstrated a high Harmonic Mean, indicating it effectively mitigates the common bias toward seen classes in realistic testing environments.

Conclusion

By reframing feature generation as a residual-learning task anchored to class prototypes, AFRNet successfully reduces class overlap in the visual feature space. The inclusion of a Feature Selection Strategy further ensures that only the most reliable semantic-to-visual mappings are utilized. While performance gains on extremely diverse datasets like SUN remain a target for future optimization, AFRNet establishes a new benchmark for robust and compact feature synthesis in Zero-Shot Learning.

Summary updated

BibTeX
@inproceedings{liu2020zero,
  author = {Bo Liu and Qiulei Dong and Zhanyi Hu},
  title = {{Zero-Shot Learning from Adversarial Feature Residual to Compact Visual Feature}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6821},
}
Figure