Adaptive Label Smoothing

Ujwal Krothapalli, A. Lynn AbbottarXiv 2020 · OpenReview

TL;DR

This paper introduces "adaptive label smoothing," a novel training technique for Convolutional Neural Networks (CNNs) that uses an object's size within an image to dynamically adjust label smoothing. This approach significantly reduces overconfidence in predictions, especially when presented with context-only images, and improves performance on downstream tasks like object detection.

What Problem Does This Paper Address?

Modern Convolutional Neural Networks (CNNs) are powerful image classifiers, but they often suffer from overconfidence in their predictions. This means they can be highly certain about incorrect guesses, which is a critical issue for safety-critical applications. A key reason for this overconfidence is how CNNs are trained. Standard training methods often use random image crops as data augmentation. While this helps generalization, it can lead to training samples that are mostly background or context, with little to no actual object. This forces the CNN to learn to rely on context rather than the object itself, leading to a bias. Furthermore, existing training methods don't account for the relative size of an object within an image, which can also contribute to misinterpretations and overconfidence.

Methods

The core innovation of this paper is adaptive label smoothing. Traditional training uses "hard labels" (a one-hot vector where the correct class is 1 and others are 0). Label smoothing, a prior technique, softens these labels by assigning a small probability to incorrect classes. This paper's approach takes label smoothing a step further by making the smoothing factor adaptive.

Here's how it works:

  • Objectness Measure: The authors leverage "objectness" scores, which quantify the likelihood that a region in an image contains an object. This is typically derived from bounding box annotations.
  • Adaptive Smoothing Factor: Instead of a fixed smoothing factor, they compute an adaptive factor based on the relative size of the object in the image. This is done by calculating the ratio of the object's bounding box area to the image area.
  • Grounding Confidence: This adaptive factor is then used to adjust the soft labels. The idea is that if an object occupies a larger portion of the image, it should contribute more to the confidence score. Conversely, if the object is small or barely present, the confidence should be lower.
  • Context-Only Handling: For images that are context-only (e.g., a dog park without dogs), the adaptive approach aims to produce high-entropy (low-confidence) predictions, reflecting the uncertainty.

The authors also explore a weighted combination of adaptive label smoothing and hard labels, controlled by a parameter $\beta$, to fine-tune the degree of context dependence.

Datasets

The research utilizes several datasets for training and evaluation:

  • ImageNet-1K: This is the primary dataset used for training and evaluating the classification performance of their adaptive label smoothing approach. They use different subsets, including one with bounding box annotations to compute objectness.
  • ImageNetV2: Used as a validation set, specifically the "MatchedFrequency" set, for evaluating classification and calibration.
  • MS COCO: Employed for transfer learning experiments, specifically for object detection tasks, to assess how well the learned representations generalize.
  • OpenImages: Another dataset used for training and evaluation, particularly for exploring performance on datasets with different characteristics and potential imbalances.

Key Insights and Ideas

  • Object Size Matters for Confidence: The most significant insight is that an object's relative size in an image is a crucial factor for determining a reliable confidence score. Ignoring this can lead to overconfidence, especially when context is misleading.
  • Adaptive Smoothing is Key: Simply using label smoothing isn't enough. Making the smoothing factor adaptive based on object size is what truly grounds the CNN's confidence in the visual evidence.
  • Contextual Bias is a Real Problem: The paper strongly highlights how random cropping, a common augmentation, can inadvertently train models to rely on context, which is detrimental for safety-critical applications. Their method directly combats this.
  • Explainable AI Through Objectness: By linking confidence to object size via objectness, the approach makes predictions more explainable. The confidence is no longer an abstract number but is tied to a tangible visual property.
  • Improved Transfer Learning: The representations learned through adaptive label smoothing show improved performance on downstream tasks like object detection, suggesting that better calibration during classification leads to more robust feature learning.

Key Results

  • Reduced Overconfidence: The adaptive label smoothing approach achieved an order of magnitude reduction in average confidence on context-only images compared to baselines like hard labels and standard label smoothing.
  • Improved Calibration: The method consistently showed lower overconfidence scores and better calibration metrics (like ECE and MCE) across various datasets (ImageNet, OpenImages).
  • Transfer Learning Gains: On the MS COCO dataset, fine-tuning models trained with adaptive label smoothing resulted in a 2.1 mAP improvement over the hard label approach.
  • Qualitative Improvements: Class Activation Maps (CAMs) demonstrated that the adaptive method localizes objects more accurately and produces higher-entropy predictions on context-only images, indicating better understanding.

Limitations and Future Work

  • Dataset Dependence: The method relies on bounding box annotations to compute objectness. While they show it works on ImageNet and OpenImages, its effectiveness might depend on the availability and quality of such annotations in other datasets.
  • Computational Cost: While not explicitly detailed as a limitation, adaptive label smoothing likely adds some computational overhead during training due to the objectness calculation.
  • Future Work: The authors mention extending their work to out-of-distribution detection, suggesting that their approach to grounding confidence might be beneficial in identifying novel or unseen data. They also aim to further investigate how adaptive label smoothing can improve decision-making systems in safety-critical applications.

Summary updated

Abstract

This paper concerns the use of objectness measures to improve the calibration performance of Convolutional Neural Networks (CNNs). CNNs have proven to be very good classifiers and generally localize objects well; however, the loss functions typically used to train classification CNNs do not penalize inability to localize an object, nor do they take into account an object's relative size in the given image. During training on ImageNet-1K almost all approaches use random crops on the images and this transformation sometimes provides the CNN with background only samples. This causes the classifiers to depend on context. Context dependence is harmful for safety-critical applications. We present a novel approach to classification that combines the ideas of objectness and label smoothing during training. Unlike previous methods, we compute a smoothing factor that is \emph{adaptive} based on relative object size within an image. This causes our approach to produce confidences that are grounded in the size of the object being classified instead of relying on context to make the correct predictions. We present extensive results using ImageNet to demonstrate that CNNs trained using adaptive label smoothing are much less likely to be overconfident in their predictions. We show qualitative results using class activation maps and quantitative results using classification and transfer learning tasks. Our approach is able to produce an order of magnitude reduction in confidence when predicting on context only images when compared to baselines. Using transfer learning, we gain 2.1mAP on MS COCO compared to the hard label approach.
BibTeX
@misc{krothapalli2020adaptive,
  author = {Ujwal Krothapalli and A. Lynn Abbott},
  title = {{Adaptive Label Smoothing}},
  year = {2020},
  eprint = {2009.06432},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.06432},
}
Figure