Normalization Matters in Zero-Shot Learning

Ivan Skorokhodov, Mohamed Elhoseiny

Introduction

Zero-Shot Learning (ZSL) aims to recognize objects from classes unseen during training by leveraging semantic descriptions, such as attributes. While normalization techniques like BatchNorm are standard in deep learning, their application in ZSL has often relied on heuristics. This paper provides a theoretical foundation for these "tricks" and introduces Class Normalization (CN), a technique that enables simple neural networks to outperform complex state-of-the-art models.

The authors identify three critical challenges in ZSL:

  1. Theoretical Gap: Widely used practices like attribute normalization and scaled cosine similarity lack rigorous explanation.
  2. Deep Model Instability: Heuristics that stabilize linear models often fail in deeper architectures.
  3. Irregular Loss Surfaces: ZSL optimization landscapes are significantly more "bumpy" than those of standard supervised classifiers.

Methodology

The researchers address these issues through variance propagation theory and architectural innovation.

Variance Analysis and Scaling

The paper demonstrates that common ZSL tricks are essentially mechanisms for preserving signal variance. They derive a formula for the optimal scaling factor ($\gamma$) in cosine similarity based on feature space dimensionality, ensuring output logit variance remains near 1.0 to prevent gradient vanishing.

Class Normalization (CN)

The primary contribution, Class Normalization, standardizes attribute embeddings class-wise before the final projection. By ensuring constant-norm inputs to the last layer and pairing this with variance-preserving weight initialization, the model achieves significantly better stability.

Continual ZSL (CZSL)

The authors also propose a framework for Continual ZSL, where models must learn a sequence of tasks without forgetting previously learned unseen classes.

Key Insights: Smoothing the Optimization Landscape

A major finding is that ZSL models receive $K$ times more gradient signals per example than standard classifiers (where $K$ is the number of classes), leading to highly irregular loss surfaces. Class Normalization acts as a smoother, bringing the optimization landscape closer to that of traditional supervised learning.

Gradient norms during training for traditional vs. ZSL models.

Experimental Results

The proposed "vanilla" 3-layer MLP with Class Normalization was evaluated on SUN, CUB, AwA1, and AwA2 benchmarks.

Performance and Efficiency

Despite its simplicity, the CN-enhanced MLP achieved state-of-the-art results on three of the four datasets. Most notably, it offers a massive efficiency gain: training takes approximately 30 seconds, compared to the 1.5 to 3 hours required by complex generative models like GANs or VAEs.

Performance comparison and training times across standard ZSL benchmarks.

Continual Learning Impact

In the CZSL benchmark, Class Normalization improved the performance of existing algorithms (e.g., EWC, A-GEM) by up to 100% in certain metrics, highlighting its importance for lifelong learning in semantic spaces.

Conclusion

This work demonstrates that proper normalization is often more critical than model complexity in ZSL. While simple MLPs with CN show remarkable performance, the gap between ZSL and fully supervised learning remains significant. Future research may focus on more sophisticated attribute-to-visual mappings that maintain these normalization benefits while capturing deeper semantic relationships.

Summary updated

Abstract

An ability to grasp new concepts from their descriptions is one of the key features of human intelligence, and zero-shot learning (ZSL) aims to incorporate this property into machine learning models. In this paper, we theoretically investigate two very popular tricks used in ZSL: "normalize+scale" trick and attributes normalization and show how they help to preserve a signal's variance in a typical model during a forward pass. Next, we demonstrate that these two tricks are not enough to normalize a deep ZSL network. We derive a new initialization scheme, which allows us to demonstrate strong state-of-the-art results on 4 out of 5 commonly used ZSL datasets: SUN, CUB, AwA1, and AwA2 while being on average 2 orders faster than the closest runner-up. Finally, we generalize ZSL to a broader problem -- Continual Zero-Shot Learning (CZSL) and test our ideas in this new setup. The source code to reproduce all the results is available at https://github.com/universome/czsl.
BibTeX
@inproceedings{skorokhodov2020normalization,
  author = {Ivan Skorokhodov and Mohamed Elhoseiny},
  title = {{Normalization Matters in Zero-Shot Learning}},
  booktitle = {King},
  year = {2020},
}
Figure