Intra-Class Uncertainty Loss Function for Classification

He Zhu, Shan Yu

Abstract

Most classification models can be considered as the process of matching templates. However, when intra-class uncertainty/variability is not considered, especially for datasets containing unbalanced classes, this may lead to classification errors. To address this issue, we propose a loss function with intra-class uncertainty following Gaussian distribution. Specifically, in our framework, the features extracted by deep networks of each class are characterized by independent Gaussian distribution. The parameters of distribution are learned with a likelihood regularization along with other network parameters. The means of the Gaussian play a similar role as the center anchor in existing methods, and the variance describes the uncertainty of different classes. In addition, similar to the inter-class margin in traditional loss functions, we introduce a margin to intra-class uncertainty to make each cluster more compact and reduce the imbalance of feature distribution from different categories. Based on MNIST, CIFAR, ImageNet, and Long-tailed CIFAR analyses, the proposed approach shows improved classification performance, through learning a better class representation.

Introduction

Standard deep learning classifiers typically operate as template-matching systems. The network learns a single "anchor" or representative point for each category in the feature space and attempts to minimize the distance between image features and their respective class anchors. However, this "one point per class" paradigm fails to account for intra-class uncertainty.

In real-world scenarios, categories exhibit varying degrees of diversity, and datasets are frequently imbalanced. A standard Softmax loss may misclassify a sample that lies between two anchors if it ignores the global distribution of the classes. For instance, a sample might be geometrically closer to Anchor A, but logically belong to Class B if Class B has a much wider distribution (variance). This lack of distribution awareness leads to network confusion, particularly in long-tailed data scenarios.

Conceptual comparison between normal classification (a) and classification considering intra-class uncertainty (b).

Methodology: The ICU Loss

The proposed Intra-Class Uncertainty (ICU) Loss models each class as an independent Gaussian distribution defined by a mean ($\mu$) and a variance ($\Sigma$). This approach shifts the decision metric from simple Euclidean distance to a Mahalanobis-based distance that incorporates the "stretch" of the class distribution.

The framework introduces a Dual-Margin Mechanism:

  1. Inter-class margin ($\alpha$): A standard technique to increase separation between different categories.
  2. Intra-class margin ($\gamma$): A novel addition that specifically constrains the uncertainty of the ground-truth class, encouraging the network to learn more compact representations.

To ensure stable training, the authors implemented a Stabilized Regularization term ($L_{reg}$) based on moment estimation, preventing the variance parameters from diverging during the optimization process.

Experiments

The ICU Loss was evaluated across several benchmarks:

  • MNIST: Used for 2-D and 100-D feature visualization.
  • CIFAR-10 & CIFAR-100: Standard image recognition tasks.
  • ImageNet: Large-scale classification using ResNet-101.
  • Long-tailed CIFAR-10/100: Testing robustness on imbalanced datasets with varying imbalance ratios.

Results and Discussion

ICU Loss consistently outperformed standard Softmax and advanced alternatives like Center Loss and L-GM Loss. On ImageNet, ICU achieved a Top-1 accuracy of 77.10%, surpassing the baseline (76.50%).

The method's strengths are most evident in Long-Tailed Performance. On CIFAR-100-LT (imbalance ratio of 100), ICU reached 39.8% accuracy, outperforming specialized methods like Focal Loss and Class-Balanced (CB) Loss.

Visualization of feature distributions on MNIST for different loss functions. ICU Loss (d, h) produces tighter, more distinct clusters compared to Softmax (a, e) or Center Loss (b, f).

As shown in the visualizations, ICU Loss produces significantly more compact and distinct feature clusters. Unlike Center Loss, which can leave "trails" of features prone to misclassification, ICU Loss "softens" the centroid constraint. This prevents overfitting to the dense centers of frequent classes and allows for better generalization on unseen test data.

Conclusion

The ICU Loss demonstrates that treating class variance as a valuable prior rather than noise significantly improves classification robustness. By modeling classes as distributions and employing a dual-margin strategy, the model effectively handles dataset imbalance and intra-class variability. Future work may extend this uncertainty-aware approach to tasks such as face verification and object detection in noisy environments.

Summary updated

Abstract

Most classification models can be considered as the process of matching templates. However, when intra-class uncertainty/variability is not considered, especially for datasets containing unbalanced classes, this may lead to classification errors. To address this issue, we propose a loss function with intra-class uncertainty following Gaussian distribution. Specifically, in our framework, the features extracted by deep networks of each class are characterized by independent Gaussian distribution. The parameters of distribution are learned with a likelihood regularization along with other network parameters. The means of the Gaussian play a similar role as the center anchor in existing methods, and the variance describes the uncertainty of different classes. In addition, similar to the inter-class margin in traditional loss functions, we introduce a margin to intra-class uncertainty to make each cluster more compact and reduce the imbalance of feature distribution from different categories. Based on MNIST, CIFAR, ImageNet, and Long-tailed CIFAR analyses, the proposed approach shows improved classification performance, through learning a better class representation.
BibTeX
@inproceedings{zhu2021intra,
  author = {He Zhu and Shan Yu},
  title = {{Intra-Class Uncertainty Loss Function for Classification}},
  booktitle = {ICME},
  year = {2021},
  url = {https://openreview.net/forum?id=mSutn-vLxZ},
}
Figure