Gaussian Affinity for Max-Margin Class Imbalanced Learning

Munawar Hayat, Salman H. Khan, Syed Waqas Zamir, Jianbing Shen, Ling Shao

Class imbalance remains a significant hurdle in deep learning, particularly in domains like medical imaging where rare conditions are underrepresented. Standard models trained with Softmax loss often prioritize majority classes, causing minority classes to be compressed into skewed regions of the feature space. This paper, presented at ICCV 2019, introduces Affinity Loss, a novel objective function designed to ensure that all classes—regardless of frequency—occupy distinct and well-defined territories in the feature space.

The Geometric Approach to Imbalance

Traditional Softmax loss relies on angular similarity (dot products), which lacks an explicit mechanism to enforce margins between classes. In imbalanced scenarios, majority classes naturally expand, "squeezing" the representation of rare classes.

Affinity Loss addresses this by shifting the optimization to Euclidean space. The method centers on three core components:

  1. Gaussian Affinity: Instead of angular alignment, the model measures the squared $\ell_2$ distance between a feature vector and a learned "class prototype" using a Gaussian similarity metric.
  2. Max-Margin Constraint: A hinge-loss penalty ensures that a sample's similarity to its ground-truth class is higher than its similarity to any other class by a predefined margin $\lambda$.
  3. Diversity Regularizer: This term forces class prototypes to be uniformly spaced and equidistant, preventing majority classes from crowding the feature space.
Geometric representation of Affinity Loss showing class prototypes ($w_A, w_B, w_C$), margins, and inter-class distances.

Experimental Results

The researchers evaluated Affinity Loss across diverse datasets, including the Dermofit Image Library (DIL) for skin lesions and the VGG2 dataset for face recognition.

Balanced Feature Representations

Visualizations of the learned feature space reveal the stark difference between standard approaches and Affinity Loss. While Softmax produces "floral petal" distributions where minority classes are thin and poorly separated, Affinity Loss generates clean, compact, and uniformly sized clusters for every category. This geometric regularity is key to the model's improved generalization on rare data.

Comparison of feature space distributions: (a) Softmax, (b) Clustering+Softmax, and (c) Affinity Loss, showing more distinct and balanced clusters for minority classes in (c).

Performance Benchmarks

  • Skin Lesion Classification: On the DIL dataset, Affinity Loss achieved 91.1% accuracy, representing a 10.9% absolute improvement over the previous state-of-the-art (CoSen CNN).
  • Robustness to Noise: The model demonstrated superior resilience to label noise. When labels were intentionally corrupted, Affinity Loss maintained significantly higher accuracy than Softmax, suggesting it learns robust underlying structures rather than simple label memorization.
  • Face Recognition: Despite being a general-purpose loss, it achieved competitive results on high-stakes benchmarks, reaching 99.65% on LFW and 97.3% on YTF, outperforming specialized losses like Triplet Loss.

Conclusion and Future Directions

Affinity Loss successfully bridges the gap between classification and clustering. By enforcing a uniform geometric structure on the feature space, it prevents the model from becoming biased toward frequent classes. While the method requires careful tuning of hyperparameters like the cluster spread ($\sigma$) and margin ($\lambda$), it offers a powerful framework for robust learning in the presence of extreme class imbalance. Future work may explore automated methods for determining the optimal number of prototypes per class to better capture internal class variance.

Summary updated

Abstract

Real-world object classes appear in imbalanced ratios. This poses a significant challenge for classifiers which get biased towards frequent classes. We hypothesize that improving the generalization capability of a classifier should improve learning on imbalanced datasets. Here, we introduce the first hybrid loss function that jointly performs classification and clustering in a single formulation. Our approach is based on an 'affinity measure' in Euclidean space that leads to the following benefits: (1) direct enforcement of maximum margin constraints on classification boundaries, (2) a tractable way to ensure uniformly spaced and equidistant cluster centers, (3) flexibility to learn multiple class prototypes to support diversity and discriminability in feature space. Our extensive experiments demonstrate the significant performance improvements on visual classification and verification tasks on multiple imbalanced datasets. The proposed loss can easily be plugged in any deep architecture as a differentiable block and demonstrates robustness against different levels of data imbalance and corrupted labels.
BibTeX
@inproceedings{hayat2019gaussian,
  author = {Munawar Hayat and Salman H. Khan and Syed Waqas Zamir and Jianbing Shen and Ling Shao},
  title = {{Gaussian Affinity for Max-Margin Class Imbalanced Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00657},
}
Figure