Co-Representation Network for Generalized Zero-Shot Learning

Fei Zhang, Guangming ShiICML 2019

Introduction: The Bias Problem in GZSL

Zero-Shot Learning (ZSL) aims to classify images of classes a model has never seen during training by bridging the gap between visual features and semantic descriptions (like attributes or word vectors). Generalized ZSL (GZSL) is a more realistic and challenging variant where the model must classify both seen and unseen classes at test time.

The fatal flaw of classic ZSL models in the GZSL setting is the bias problem. Because the mapping from the semantic space to the visual space is trained exclusively on seen classes, the model overfits. In the resulting embedding space, the projected features of unseen classes end up clumped too close to those of seen classes. Consequently, the model exhibits a massive bias, misclassifying almost all unseen images into known, seen categories.

Recently, researchers have turned to complex generative models (like GANs and VAEs) to synthesize unseen data and balance the scales. This paper asks a different question: Can we solve the bias problem with a simpler, traditional embedding architecture? The authors introduce the Co-Representation Network (CRnet), a surprisingly simple embedding model that uses parallel "expert" modules to learn a highly locally linear, uniform visual embedding space.

Methodology: The Co-Representation Network

To tackle the bias problem, the authors first propose a new metric called Local Relative Distance (LRD) to quantify how well unseen class anchors are separated from seen class anchors in the embedding space.

They then introduce CRnet, which consists of two main components:

CRnet Architecture
  1. Cooperation Module (The Novelty): Instead of using a deep, complex neural network to project semantic vectors into the visual space, CRnet uses a "divide and conquer" approach. It clusters the semantic space into $K$ fields using K-means. Each field is assigned an "expert module"—a simple single-layer perceptron with a ReLU activation. The final projection is the sum of these parallel experts. This creates a piecewise linear projection.
  2. Relation Module (The Classifier): Adopted from prior work, this is a two-layer multi-layer perceptron (MLP) that learns a data-driven, non-linear similarity metric between the projected semantic anchors and the actual visual features, replacing rigid metrics like cosine similarity.

The entire network is trained end-to-end using Mean Squared Error (MSE) loss, with specific regularization terms to prevent overfitting in both modules.

Key Insights

  • High Local Linearity Reduces Bias: The core mathematical insight of the paper is that highly non-linear projections cause the embedding space to collapse, exacerbating the bias problem. By using parallel single-layer experts with ReLU activations, CRnet forces the projection to be piecewise linear. The authors prove mathematically that this high local linearity ensures unseen class anchors maintain a larger distance (higher LRD) from seen classes.
  • Sparsity Equals Discriminability: Visualizing the feature anchors reveals that CRnet produces much sparser representations compared to standard deep MLPs. This sparsity means the network relies on fewer, more discriminative features, avoiding the "over-confidence" phenomenon where networks memorize redundant features specific only to the training (seen) set.
Sparse Feature Representations
  • Initialization Matters: To ensure the parallel expert modules don't just learn the exact same weights and collapse into a single function, the K-means initialization is crucial. It forces different experts to specialize in different geometric regions of the semantic space.

Results and Limitations

The authors evaluated CRnet using standard GZSL splits and 2048-dimensional ResNet-101 extracted features on five popular benchmark datasets: AwA1, AwA2, aPY, CUB, and SUN.

State-of-the-Art Performance: CRnet achieved the highest Harmonic Mean (H) accuracy on 4 out of 5 datasets: AwA1 (65.4%), AwA2 (63.1%), CUB (50.5%), and aPY (44.0%). It outperformed both classic embedding models and recent complex generative models (like SE-GZSL and f-CLSWGAN).

Massive Gains on Unseen Classes: Compared to a baseline Relation Network (RN) that uses a standard MLP for projection, CRnet drastically improved unseen class accuracy. For example, on AwA2, CRnet achieved 52.6% unseen accuracy compared to the baseline's 30.0%. Furthermore, the authors calculated the average LRD for the networks and found CRnet consistently produced higher LRD scores than the baseline, proving its embedding space is indeed more uniform and less biased.

Limitations: The authors note a fundamental limitation of embedding models compared to generative ones. Because CRnet relies directly on the learned projection rather than synthesizing a balanced dataset, its unseen accuracy is strictly bottlenecked by its seen accuracy. On the SUN dataset, where seen accuracy is inherently low (36.5%), CRnet's unseen accuracy suffers (34.1%), allowing generative models to outperform it on this specific benchmark.

Despite this, the core mechanism—using a parallel, cooperative structure to enforce high local linearity—proves highly effective for alleviating bias and overfitting, suggesting potential applications in broader transfer learning tasks beyond just GZSL.

Summary updated

BibTeX
@inproceedings{zhang2019co,
  author = {Fei Zhang and Guangming Shi},
  title = {{Co-Representation Network for Generalized Zero-Shot Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
}
Figure