Hyperspherical Prototype Networks

Pascal Mettes, Elise van der Pol, Cees Snoek

In the world of deep learning, neural networks are typically left to figure out their own internal representations. We provide a target—such as a one-hot vector for classification or a scalar for regression—and let backpropagation do the heavy lifting. But what if we pre-organized the output space before training even began?

In their NeurIPS 2019 paper, "Hyperspherical Prototype Networks," Pascal Mettes and colleagues from the University of Amsterdam propose a paradigm shift: instead of letting class "prototypes" drift around during training, we should fix them on a hypersphere a priori.

This framework unifies classification and regression by mapping outputs to a hypersphere with fixed prototypes. By pre-arranging these prototypes for maximum separation, the authors create a more robust, flexible, and computationally efficient training process that excels in low-data and multi-task scenarios.

The "Moving Target" Problem

Traditional deep learning models treat classification and regression as fundamentally different tasks. Classification usually relies on softmax cross-entropy, forcing the output to match a specific number of classes. Regression, on the other hand, typically uses Mean Squared Error (MSE) on a single-dimensional output.

Existing "prototype-based" networks (like Prototypical Networks for few-shot learning) attempt to bridge this gap by defining a class by its "mean" position in the output space. However, this creates a "moving target" problem: as the network learns, the mean position of the classes shifts, requiring constant re-estimation. Furthermore, these methods often struggle with large-scale datasets or fail to generalize well to regression tasks.

Methods: The Hypersphere Output Space

The core innovation of this paper is utilizing a hypersphere (a sphere in $D$ dimensions) as the universal output space for both classification and regression.

Hyperspherical output spaces for classification (left) and regression (right).
  1. Fixed Prototypes: Before training begins, the authors distribute $K$ prototypes (one for each class) as uniformly as possible across the hypersphere. This is a classic mathematical challenge known as the Tammes problem. Because an exact solution is difficult for high dimensions, they use a differentiable loss function to "push" prototypes away from each other until they are maximally spread out.
  2. Cosine Similarity Loss: Instead of cross-entropy, the network is trained to maximize the cosine similarity between its output vector and the fixed prototype of the correct class.
  3. Privileged Information: The output space can be further refined using word embeddings (like word2vec). This ensures that semantically similar classes (e.g., "cat" and "dog") are placed closer together on the sphere than unrelated ones (e.g., "cat" and "car").
  4. Regression as Interpolation: For regression tasks, the authors place two prototypes at opposite poles of the sphere (representing the minimum and maximum values). The network then learns to project outputs at specific angles between these poles.
  5. Multi-task Learning: Because classification and regression now share the exact same loss function and output space, they can be trained jointly without the usual headache of "weighting" different loss terms.

Key Insights and Advantages

  • The Power of Inductive Bias: By forcing the network to respect large-margin separation from the start, the model learns more efficiently. It doesn't have to "discover" that classes should be far apart; it is a fundamental rule of the output space.
  • Dimensionality Flexibility: Unlike softmax, which requires the output layer to match the number of classes exactly, Hyperspherical Prototype Networks can use any dimensionality. A model can compress 100 classes into a 10-dimensional space and still maintain high accuracy.
  • Robustness to Imbalance: The fixed-prototype approach is significantly more stable than softmax when some classes have very few examples. In softmax, a rare class might get "pushed around" by the gradients of dominant classes; here, its location on the hypersphere is permanent.
  • Regression Degrees of Freedom: By performing regression on a hypersphere rather than a 1D line, the model has more "room" to organize its internal logic, leading to more robust predictions.

Empirical Results

The authors evaluated their approach across a diverse set of benchmarks, including CIFAR-100, ImageNet-200, and CUB Birds 200-2011 for classification, and OmniArt for regression and joint tasks.

Training efficiency of Hyperspherical Prototype Networks compared to moving-target baselines.
  • Superior Efficiency: As shown in the training curves, the hyperspherical approach converges faster and achieves higher test accuracy than moving-target baselines like Deep NCM. On CIFAR-100, it outperformed word2vec-based baselines by over 7 percentage points and maintained high accuracy even when the output dimensionality was slashed by 75%.
  • Better than Softmax in Long-Tail Scenarios: When some classes have very few samples, the proposed method outperformed standard softmax cross-entropy (46.4% vs 44.2% on a modified CIFAR-100).
  • Regression Gains: On the OmniArt year-prediction task, the hyperspherical method achieved a Mean Absolute Error (MAE) of 73.2, significantly better than the standard MSE baseline of 109.9.
  • Seamless Multi-tasking: In joint tasks (like identifying a rotated MNIST digit while predicting its rotation angle), the model outperformed standard multi-task baselines across all task-weight settings, proving that a shared output space is a powerful tool for learning related concepts.

Limitations and Future Work

While highly effective, the method currently relies on a fixed vocabulary. If a new class needs to be added later, the prototype distribution optimization must be re-run (though the entire network may not need full retraining). The authors suggest that future work could explore incremental learning (adding classes on the fly) and open-set recognition, where the model must identify if an object belongs to any of the known prototypes or if it is something entirely new.

Reference: Mettes, P., van der Pol, E., & Snoek, C. G. M. (2019). Hyperspherical Prototype Networks. Advances in Neural Information Processing Systems (NeurIPS).

Summary updated

Abstract

This paper introduces hyperspherical prototype networks, which unify classification and regression with prototypes on hyperspherical output spaces. For classification, a common approach is to define prototypes as the mean output vector over training examples per class. Here, we propose to use hyperspheres as output spaces, with class prototypes defined a priori with large margin separation. We position prototypes through data-independent optimization, with an extension to incorporate priors from class semantics. By doing so, we do not require any prototype updating, we can handle any training size, and the output dimensionality is no longer constrained to the number of classes. Furthermore, we generalize to regression, by optimizing outputs as an interpolation between two prototypes on the hypersphere. Since both tasks are now defined by the same loss function, they can be jointly trained for multi-task problems. Experimentally, we show the benefit of hyperspherical prototype networks for classification, regression, and their combination over other prototype methods, softmax cross-entropy, and mean squared error approaches.
BibTeX
@inproceedings{mettes2019hyperspherical,
  author = {Pascal Mettes and Elise van der Pol and Cees Snoek},
  title = {{Hyperspherical Prototype Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure