Rethinking Generative Zero-Shot Learning: An Ensemble Learning Perspective for Recognising Visual Patches

Zhi Chen, Sen Wang, Jingjing Li, Zi Huang

Abstract

Generative Zero-Shot Learning (ZSL) aims to recognize unseen object classes by synthesizing visual features from semantic descriptions. However, existing methods often suffer from background noise in global features and a lack of focus on discriminative local details. This paper introduces Multi-Patch Generative Adversarial Nets (MPGAN), a framework that adopts a "divide-and-conquer" approach. By training dedicated generators for specific semantic patches and combining their outputs through a novel weighted ensemble strategy, MPGAN achieves state-of-the-art performance in fine-grained classification tasks.

Introduction: The Challenges of Generative ZSL

Zero-Shot Learning (ZSL) is a critical frontier in computer vision, enabling models to identify objects without prior training examples by leveraging auxiliary information like text descriptions. Despite progress, generative ZSL models face two primary hurdles:

  1. Background Noise: Most models synthesize a single "global" feature for an entire image. This process inadvertently incorporates irrelevant background elements (e.g., sky, foliage), which can mislead the final classifier.
  2. Feature Uniformity: In fine-grained recognition—such as distinguishing between similar bird species—discriminative features are often localized (e.g., the specific shape of a beak). Standard models treat all image regions with equal weight, failing to capture these subtle "make-or-break" characteristics.
Comparison of global image features versus localized semantic patch features.

Methods: The MPGAN Framework

To address these issues, the authors propose Multi-Patch Generative Adversarial Nets (MPGAN). The core philosophy is to break down the complex task of global feature generation into specialized sub-tasks.

Multi-Patch Generation

Instead of a single generalist generator, MPGAN decomposes an image into semantic patches (e.g., head, breast, wing, tail). A dedicated GAN is assigned to each patch. For instance, a "head GAN" focuses exclusively on synthesizing features for the head region. This specialization leads to higher-quality, more robust representations that are inherently less susceptible to global background noise.

Evolution of generative ZSL architectures from (a) Global generation to (c) the proposed Multi-patch generation (MPGAN).

Denoising and Regularization

Semantic inputs, such as Wikipedia articles, often contain information irrelevant to specific visual patches. MPGAN employs a denoising module (using Fully Connected layers or PCA) to filter text embeddings before they reach the generators. Additionally, a visual pivot regularization term is included in the loss function. This pushes generated samples toward the actual visual center of a class, ensuring the synthesized features remain realistic.

Ensemble Voting with Discrimination-based Attention

Each patch-specific generator is paired with its own classifier. To aggregate these predictions, the authors developed a novel attention mechanism based on an offline algorithm. Rather than using a complex neural network, weights are calculated based on the inter-class vs. intra-class distance ratio of features in the training set. Patches that demonstrate higher discriminative power on seen data (like the head) receive higher weights during the final inference for unseen classes.

Experimental Results

MPGAN was evaluated on two major fine-grained bird datasets: CUB (Caltech-UCSD Birds 200) and NAB (North America Birds). Unlike many studies that use clean attributes, this work utilized noisy TF-IDF representations from Wikipedia.

  • State-of-the-Art Accuracy: On the CUB dataset, MPGAN achieved 48.2% Top-1 accuracy, surpassing previous benchmarks. On the larger NAB dataset, it reached 41.2%, maintaining a significant lead over other generative methods.
  • Zero-Shot Retrieval: The model demonstrated superior performance in retrieving images of unseen classes from a database based solely on text descriptions, achieving high mean Average Precision (mAP).
  • Ablation and Sensitivity: Sensitivity analysis confirmed that the "head" patch is the most discriminative for bird species. However, the ensemble approach remains vital, as the head alone provides significantly lower accuracy than the combined multi-patch system.

Conclusion and Future Work

MPGAN demonstrates that a specialized, ensemble-based approach to feature generation can effectively overcome the limitations of global generative ZSL. While currently optimized for domains with well-defined semantic patches like ornithology, future work aims to automate patch discovery. This would allow the framework to generalize to more diverse categories, such as manufactured goods or varied natural species, where predefined semantic parts may not be as obvious.

Summary updated

Abstract

Zero-shot learning (ZSL) is commonly used to address the very pervasive problem of predicting unseen classes in fine-grained image classification and other tasks. One family of solutions is to learn synthesised unseen visual samples produced by generative models from auxiliary semantic information, such as natural language descriptions. However, for most of these models, performance suffers from noise in the form of irrelevant image backgrounds. Further, most methods do not allocate a calculated weight to each semantic patch. Yet, in the real world, the discriminative power of features can be quantified and directly leveraged to improve accuracy and reduce computational complexity. To address these issues, we propose a novel framework called multi-patch generative adversarial nets (MPGAN) that synthesises local patch features and labels unseen classes with a novel weighted voting strategy. The process begins by generating discriminative visual features from noisy text descriptions for a set of predefined local patches using multiple specialist generative models. The features synthesised from each patch for unseen classes are then used to construct an ensemble of diverse supervised classifiers, each corresponding to one local patch. A voting strategy averages the probability distributions output from the classifiers and, given that some patches are more discriminative than others, a discrimination-based attention mechanism helps to weight each patch accordingly. Extensive experiments show that MPGAN has significantly greater accuracy than state-of-the-art methods.
BibTeX
@inproceedings{chen2020rethinking,
  author = {Zhi Chen and Sen Wang and Jingjing Li and Zi Huang},
  title = {{Rethinking Generative Zero-Shot Learning: An Ensemble Learning Perspective for Recognising Visual Patches}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=IgeaJ7ZIql},
}
Figure