Class Normalization for (Continual)? Generalized Zero-Shot Learning

Ivan Skorokhodov, Mohamed ElhoseinyICLR 2021 · OpenReview

TL;DR

This paper reveals that the "secret sauce" for state-of-the-art Zero-Shot Learning (ZSL) isn't necessarily a complex generative architecture, but rather rigorous signal normalization. By introducing Class Normalization (CN), the authors enable a simple Multi-Layer Perceptron (MLP) to outperform sophisticated models while training up to 100 times faster. Furthermore, they extend these benefits to a new, more challenging "Continual" ZSL setting.

The Challenge of Zero-Shot Learning

Zero-Shot Learning (ZSL) is the AI equivalent of learning from a manual. Instead of seeing thousands of pictures of a "Zebra," a model is given a semantic description (e.g., "has stripes," "looks like a horse") and must recognize one in the wild.

While the field has moved toward increasingly complex models—like Generative Adversarial Networks (GANs) that synthesize "fake" images of unseen classes—training these is slow and computationally expensive. Furthermore, ZSL practitioners often rely on "tricks" (like scaling cosine similarities) without a clear theoretical understanding of why they work. This paper tackles two primary gaps:

  1. The Theoretical Gap: Why do common ZSL "tricks" work, and why do they fail as models get deeper?
  2. The Practical Gap: How can we make ZSL models more stable and efficient, especially when learning tasks sequentially (Continual Learning)?

Methods: The Mechanics of Class Normalization

The authors start by mathematically analyzing two ubiquitous ZSL tricks: Normalize+Scale (NS) and Attributes Normalization (AN). They prove that these tricks essentially act as variance stabilizers for linear models, preventing the signal from exploding or vanishing during training.

However, they demonstrate that these tricks fail in deep, multi-layer architectures. To fix this, they propose Class Normalization (CN), which consists of two parts:

  1. Class-wise Standardization: A layer that standardizes attribute embeddings (the model's internal representation of a class description) to have zero mean and unit variance across the class dimension.
  2. Corrected Initialization: A specific variance formula for initializing the final projection layer, ensuring that the signal variance remains consistent regardless of the number of hidden units or features.
Variance and gradient norm stabilization across training iterations. Class Normalization (CN) keeps variance stable in non-linear models and smooths the optimization landscape.

As shown in the figure above, ZSL models naturally have much more irregular "loss landscapes" than standard classifiers because every training sample updates the parameters for all classes simultaneously. CN acts as a stabilizer, smoothing out this landscape, keeping variance in check, and making optimization much easier.

Key Insights and Results

The researchers evaluated their approach on four gold-standard ZSL benchmarks: SUN (Scene attributes), CUB (Birds), and AwA1 & AwA2 (Animals with Attributes).

  • Simplicity is Power: The most striking insight is that a "vanilla" 3-layer MLP, when properly normalized with CN, can beat state-of-the-art models that use complex data-generation schemes.
  • The "Why" of the Scaling Factor ($\gamma$): ZSL models often use a hyperparameter $\gamma$ to scale logits. The authors provide a theoretical formula to calculate the "optimal" $\gamma$ based on feature dimensionality, removing the need for tedious trial-and-error grid searches.
Performance comparison of the proposed Vanilla MLP with Class Normalization against state-of-the-art generative models. Note the drastic reduction in training time.
  • State-of-the-Art Performance: The CN-enhanced MLP achieved top-tier results on 3 out of 4 major datasets, outperforming complex models like DCN, SGAL, and LsrGAN on the Harmonic mean (H) metric.
  • Massive Speedup: Because the model is a simple MLP rather than a generative network, it trains in roughly 30 seconds, compared to the 50 minutes to 1.25 hours required by many modern competitors—a massive improvement in efficiency.

Bridging to Continual Zero-Shot Learning

Finally, the authors formalize Continual Zero-Shot Learning (CZSL), where a model learns new classes in a sequence of tasks without forgetting the old ones. They repurposed CUB (split into 10 sequential tasks) and SUN (split into 15 sequential tasks) to test how well models retain knowledge over time.

In this sequential task setting, simply adding Class Normalization significantly boosted the performance of existing continual learning algorithms (like EWC and A-GEM), consistently reducing "catastrophic forgetting" and improving joint accuracy across all tested baselines.

Limitations and Future Work

While Class Normalization makes simple models highly competitive, the authors acknowledge that Continual ZSL remains an incredibly difficult problem. Even with CN, there is still a significant performance gap between sequential learning and "multi-task" learning (where the model sees all data at once).

Future research could explore whether CN can be integrated into more complex backbone architectures or if specialized "memory" mechanisms can further bridge the gap in the lifelong learning scenario. Additionally, while the theoretical derivations hold for standard attribute datasets, their application to raw text descriptions (which are noisier) remains an open question.

Summary updated

Abstract

Normalization techniques have proved to be a crucial ingredient of successful training in a traditional supervised learning regime. However, in the zero-shot learning (ZSL) world, these ideas have received only marginal attention. This work studies normalization in ZSL scenario from both theoretical and practical perspectives. First, we give a theoretical explanation to two popular tricks used in zero-shot learning: normalize+scale and attributes normalization and show that they help training by preserving variance during a forward pass. Next, we demonstrate that they are insufficient to normalize a deep ZSL model and propose Class Normalization (CN): a normalization scheme, which alleviates this issue both provably and in practice. Third, we show that ZSL models typically have more irregular loss surface compared to traditional classifiers and that the proposed method partially remedies this problem. Then, we test our approach on 4 standard ZSL datasets and outperform sophisticated modern SotA with a simple MLP optimized without any bells and whistles and having ~50 times faster training speed. Finally, we generalize ZSL to a broader problem — continual ZSL, and introduce some principled metrics and rigorous baselines for this new setup. The source code is available at https://github.com/universome/class-norm.
BibTeX
@inproceedings{skorokhodov2021class,
  author = {Ivan Skorokhodov and Mohamed Elhoseiny},
  title = {{Class Normalization for (Continual)? Generalized Zero-Shot Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=Met0PDxfLKJ},
}
Figure