Awareness of uncertainty in classification using a multivariate model and multi-views

Alexey Kornaev, Elena Kornaeva, Oleg Ivanov, Ilya Pershin, Danis AlukaevarXiv 2024 · DOI · arXiv

Abstract

Standard deep learning classifiers often struggle with overconfidence, particularly when trained on noisy data. This paper presents a novel approach to uncertainty-aware classification by introducing the Uncertainty-Aware Negative Log-Likelihood (UANLL) loss function. By modeling classification as a multivariate regression task, the proposed method enables models to simultaneously predict class labels and estimate their own predictive uncertainty. Furthermore, the researchers leverage multi-view inference—aggregating predictions from multiple augmented versions of an input—to enhance robustness. Experimental results on CIFAR-10 with significant label noise demonstrate that this approach substantially outperforms traditional Cross-Entropy baselines and specialized state-of-the-art methods.

Introduction

In deep learning, classification models typically employ Cross-Entropy (CE) loss. While the resulting softmax outputs are frequently interpreted as confidence scores, they are often poorly calibrated and fail to represent true predictive uncertainty. This is especially problematic in the presence of "aleatoric" uncertainty, which arises from inherent noise in the data, such as mislabeled training samples.

In contrast, regression tasks often utilize heteroscedastic loss functions to predict both a mean and a variance, providing a natural measure of uncertainty. This paper bridges the gap between these two domains, proposing a framework where a classifier learns to "doubt" its own predictions, leading to more reliable decision-making in noisy environments.

Methodology

Uncertainty-Aware Negative Log-Likelihood (UANLL)

The core contribution is the UANLL loss function. Instead of treating classes as discrete categories, the authors model the classification task as a multivariate normal distribution with a "spherical" variance. The network is modified to output two components: a vector of class predictions ($h$) and a scalar uncertainty estimate ($s$).

The UANLL loss function is designed to penalize prediction errors while allowing the model to increase $s$ for difficult or noisy samples. By increasing $s$, the model can "explain away" high errors, though a penalty term prevents it from predicting high uncertainty for all inputs. This mechanism encourages the model to identify and flag inherently ambiguous samples during training.

The UANLL loss architecture: the model generates class predictions (h) and an uncertainty estimate (s) to compute the final loss.

Multi-View Inference and Aggregation

To further improve robustness during testing, the authors employ multi-view inference. Rather than relying on a single prediction, the model evaluates $n$ different augmented versions (e.g., random crops) of the same test image. These views are then aggregated using several strategies:

  • MVM (Majority Vote): A simple mode-based aggregation.
  • MVWCo/MVWCe: Votes are weighted by the model's internal confidence or predicted certainty.
  • Hard vs. Soft Weighting: "Hard" weighting schemes utilize a threshold to completely discard views where the model's uncertainty exceeds a certain limit. The optimal thresholds and augmentation parameters are determined using Particle Swarm Optimization (PSO).

Experimental Results

The proposed methodology was evaluated on the CIFAR-10 dataset under varying levels of label noise: clean labels, 20% asymmetric noise, and 40% asymmetric noise.

Performance under Noise

The UANLL approach demonstrated exceptional resilience to label noise. On CIFAR-10 with 40% asymmetric noise, the UANLL model with hard confidence weighting (MVWCo-H) achieved an accuracy of 75.20%. This significantly exceeds the baseline CE model (~65.15%) and outperforms specialized methods like Co-teaching (~73.43%) and S2E (~75.08%). Even on clean data, the multi-view approach boosted accuracy from a baseline of ~87% to over 93%.

Impact of Multi-Views and Thresholding

Ablation studies highlighted the importance of the number of views and the optimization of hyperparameters. As shown in the figures below, accuracy increases with the number of views before plateauing around 40-50 views. The use of uncertainty as a filter (Hard Weighting) is particularly effective in noisy scenarios, as it allows the final prediction to be dominated by "clear" views while ignoring noisy or ambiguous ones.

Preliminary experiments illustrating accuracy gains relative to (a) the number of multi-views, (b) random crop scales, and (c) optimized uncertainty thresholds.

Discussion and Conclusion

By treating classification through the lens of multivariate regression, this work provides a robust framework for uncertainty estimation. The synergy between the UANLL loss and multi-view inference allows models to effectively navigate high-noise environments.

However, some limitations remain. The current approach primarily addresses aleatoric uncertainty (data noise) rather than epistemic uncertainty (model knowledge gaps). Additionally, while the predicted variance improves performance, its direct interpretability as a probability of error requires further investigation. Future work could explore the application of these techniques to more complex domains such as object detection and out-of-distribution detection.

Summary updated

Abstract

One of the ways to make artificial intelligence more natural is to give it some room for doubt. Two main questions should be resolved in that way. First, how to train a model to estimate uncertainties of its own predictions? And then, what to do with the uncertain predictions if they appear? First, we proposed an uncertainty-aware negative log-likelihood loss for the case of N-dimensional multivariate normal distribution with spherical variance matrix to the solution of N-classes classification tasks. The loss is similar to the heteroscedastic regression loss. The proposed model regularizes uncertain predictions, and trains to calculate both the predictions and their uncertainty estimations. The model fits well with the label smoothing technique. Second, we expanded the limits of data augmentation at the training and test stages, and made the trained model to give multiple predictions for a given number of augmented versions of each test sample. Given the multi-view predictions together with their uncertainties and confidences, we proposed several methods to calculate final predictions, including mode values and bin counts with soft and hard weights. For the latter method, we formalized the model tuning task in the form of multimodal optimization with non-differentiable criteria of maximum accuracy, and applied particle swarm optimization to solve the tuning task. The proposed methodology was tested using CIFAR-10 dataset with clean and noisy labels and demonstrated good results in comparison with other uncertainty estimation methods related to sample selection, co-teaching, and label smoothing.
BibTeX
@misc{kornaev2024awareness,
  author = {Alexey Kornaev and Elena Kornaeva and Oleg Ivanov and Ilya Pershin and Danis Alukaev},
  title = {{Awareness of uncertainty in classification using a multivariate model and multi-views}},
  year = {2024},
  eprint = {2404.10314},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.10314},
}
Figure