Zero Shot Learning with the Isoperimetric Loss

Shay Deutsch, Andrea L. Bertozzi, Stefano Soatto

Abstract

We introduce the isoperimetric loss as a regularization criterion for learning the map from a visual representation to a semantic embedding, to be used to transfer knowledge to unknown classes in a zero-shot learning setting. We use a pre-trained deep neural network model as a visual representation of image data, a Word2Vec embedding of class labels, and linear maps between the visual and semantic embedding spaces. However, the spaces themselves are not linear, and we postulate the sample embedding to be populated by noisy samples near otherwise smooth manifolds. We exploit the graph structure defined by the sample points to regularize the estimates of the manifolds by inferring the graph connectivity using a generalization of the isoperimetric inequalities from Riemannian geometry to graphs. Surprisingly, this regularization alone, paired with the simplest baseline model, outperforms the state-of-the-art among fully automated methods in zero-shot learning benchmarks such as AwA and CUB. This improvement is achieved solely by learning the structure of the underlying spaces by imposing regularity.

Introduction

Zero-Shot Learning (ZSL) aims to recognize objects from classes never seen during training. This is typically achieved by mapping images into a visual embedding space and then projecting these features into a semantic space (e.g., Word2Vec or attribute lists) where unseen classes are defined. The primary challenge lies in the regularization of this mapping; without training examples for unseen classes, functions often fail to generalize, becoming noisy or overfitting to seen data.

While many existing methods rely on complex neural architectures or manual human annotations, this work explores whether superior results can be achieved simply by enforcing better geometric structure on the data. We postulate that data resides on smooth manifolds, but observed samples are corrupted by noise.

Methods: Isoperimetric Regularization

The proposed approach utilizes a simple linear mapping between spaces, augmented by a sophisticated geometric regularizer: the Isoperimetric Loss (IPL).

  1. Graph Representation: Visual embeddings are treated as vertices in a graph, with edges representing similarities between images.
  2. The Isoperimetric Principle: Drawing from classical geometry, where a sphere has the smallest surface area for a given volume, we seek "compact" clusters in the graph. This is achieved by minimizing the "flow" (edge weights) escaping a cluster boundary relative to its internal volume.
  3. Spectral Denoising: To avoid the high computational cost of direct optimization, we employ Spectral Graph Wavelets (SGW). This allows for "denoising" the graph signal in the spectral domain, effectively shrinking cluster boundaries and making them more distinct.
Visualizing the effect of IPL regularization on AWA1 attributes: (a) noisy attributes vs (b) regularized attributes using IPL.

By smoothing the manifolds, IPL restores the underlying topology of the visual-to-semantic map, allowing for more accurate spectral clustering and association with unseen class labels.

Experimental Results

The model was evaluated on standard ZSL benchmarks: Animals with Attributes (AwA1, AwA2) and Caltech-UCSD Birds (CUB). The focus was on the fully automated setting, using Word2Vec embeddings rather than human-annotated attributes.

Comparison of the proposed IPL method against state-of-the-art automated ZSL methods.

As shown in the table above, the IPL method set new records for fully automated ZSL at the time of publication:

  • AwA1: Achieved 76.03% accuracy, significantly outperforming previous automated baselines.
  • CUB: Achieved 39.4% accuracy, representing a substantial jump over standard linear mappings.
  • Generalized ZSL (GZSL): In settings involving both seen and unseen classes, IPL showed massive gains, reducing error by over 40% on the CUB dataset compared to some existing methods.

The method is also computationally efficient, regularizing over 5,000 points in approximately 45 seconds on a standard CPU.

Discussion

The success of IPL suggests that the geometric structure of data contains more transferable information than previously exploited. By prioritizing structure over model complexity, a simple linear baseline can outperform non-linear models.

Future work may address scalability to millions of points, as graph-based methods are currently memory-intensive. Additionally, integrating IPL as a general-purpose regularizer within deeper neural pipelines could further enhance performance in various transfer learning tasks.

Summary updated

Abstract

We introduce the isoperimetric loss as a regularization criterion for learning the map from a visual representation to a semantic embedding, to be used to transfer knowledge to unknown classes in a zero-shot learning setting. We use a pre-trained deep neural network model as a visual representation of image data, a Word2Vec embedding of class labels, and linear maps between the visual and semantic embedding spaces. However, the spaces themselves are not linear, and we postulate the sample embedding to be populated by noisy samples near otherwise smooth manifolds. We exploit the graph structure defined by the sample points to regularize the estimates of the manifolds by inferring the graph connectivity using a generalization of the isoperimetric inequalities from Riemannian geometry to graphs. Surprisingly, this regularization alone, paired with the simplest baseline model, outperforms the state-of-the-art among fully automated methods in zero-shot learning benchmarks such as AwA and CUB. This improvement is achieved solely by learning the structure of the underlying spaces by imposing regularity.
BibTeX
@inproceedings{deutsch2020zero,
  author = {Shay Deutsch and Andrea L. Bertozzi and Stefano Soatto},
  title = {{Zero Shot Learning with the Isoperimetric Loss}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=vAEwKTueH6},
}
Figure