Auxiliary Image Regularization for Deep CNNs with Noisy Labels Samaneh Azadi

Jiashi Feng, Stefanie Jegelka, Trevor Darrell

Training deep convolutional neural networks (CNNs) usually feels like a "garbage in, garbage out" scenario. If your training data is full of mislabeled images—a common headache when scraping data from the web—the model's performance typically plummets. To address this, researchers have proposed a clever way to make CNNs more resilient to these errors using a technique called Auxiliary Image Regularization (AIR).

By exploiting the underlying structure of the data, the AIR framework allows the model to automatically identify and "mute" mislabeled images during training. This enables the network to prioritize reliable samples, significantly outperforming standard methods on datasets with up to 50% label noise.

The Problem with Noisy Labels

Deep learning models are notoriously data-hungry, but high-quality, human-verified labels are expensive and slow to collect. The alternative—using images with user-supplied tags from the web—is cheap but "noisy." A photo tagged as "dog" might actually be a "wolf" or even a "park bench."

Standard CNNs are highly sensitive to this noise. Most existing solutions try to model a global "confusion matrix" (e.g., calculating how often a "cat" is mistaken for a "dog"). However, these methods often fail to account for the specific features of individual images. They don't look at a specific "dog" image and realize, "This looks nothing like the other dogs; maybe I should ignore its label."

Methods: Auxiliary Image Regularization (AIR)

The core innovation of this research is the AIR framework, which shifts the focus from regularizing model parameters to regularizing how the model perceives the data.

Feature manifold visualization

Here is how the framework operates:

  1. Manifold Exploitation: The model uses features extracted from a CNN pre-trained on a clean, independent dataset (like ImageNet). This provides a "stable" feature space where similar objects naturally cluster together regardless of their noisy labels.
  2. Neighbor Regularization: Intuitively, AIR acts like a "nearest neighbor" check. If an image is labeled "Blue Jay" but its features place it deep in a cluster of "Cardinals," the regularizer suppresses its influence on the "Blue Jay" classifier.
  3. Group Sparsity on Responses: Instead of the usual approach of regularizing the model's weights (like L2 regularization), AIR regularizes the response maps of the top layer. It treats each image as a "group" and applies a group-sparse norm ($l_{1,2}$-norm). This encourages the model to set the weights of "unreliable" images to zero.
  4. Stochastic ADMM: Because the group-norm is non-smooth, standard backpropagation via Stochastic Gradient Descent (SGD) struggles to solve it efficiently. The authors instead use the Stochastic Alternating Direction Method of Multipliers (ADMM), which is much better suited for this type of optimization.

Key Insights and Ideas

  • Data-Centric Regularization: The shift from regularizing parameters to regularizing how the model sees the data is a powerful paradigm. It allows the model to actively select which training samples to trust.
  • The Power of Pre-training: The paper demonstrates that a manifold learned on one clean dataset (e.g., general ImageNet) is robust enough to serve as a "sanity check" for a completely different noisy task.
  • Optimization Matters: A key insight is the comparison between SGD and ADMM. For this specific regularizer, ADMM achieved 72% accuracy on CIFAR-10 compared to just 66% for SGD, proving that the choice of optimizer is critical when dealing with non-smooth constraints.

Key Results

The authors evaluated AIR using both synthetic noise (subsets of ImageNet7k, MNIST, and CIFAR-10 with manually flipped labels) and real-world noise (NUS-WIDE-LITE, containing Flickr images with messy user tags).

Accuracy results across noise levels
  • Superior Robustness: On the ImageNet7k subset with 50% noise, AIR achieved 61.5% accuracy, significantly outperforming a standard SVM (53.44%) and Robust Logistic Regression (58.71%).
  • Resilience to Heavy Noise: As the percentage of incorrect labels increased from 30% to 50%, the performance of standard models dropped sharply, while AIR remained remarkably stable and consistently outperformed baselines.
  • Qualitative Filtering: Visualizations of the model's behavior show that AIR successfully assigns the lowest "activity" scores to images that are clearly mislabeled or contain multiple conflicting objects, effectively filtering them out of the gradient update.

Limitations and Future Work

While highly effective, AIR currently relies on a well-trained deep model on an auxiliary clean dataset to establish the initial manifold. If no such clean dataset exists for a highly specialized domain, the method might struggle to find reliable feature clusters. Furthermore, in this work, the regularizer was only applied to the final classification layer.

The authors suggest moving toward an end-to-end scheme in the future. By allowing the error information from the auxiliary regularizer to backpropagate through the entire network, the model could potentially refine the feature representations themselves to be inherently noise-aware.

Summary updated

Abstract

Precisely-labeled data sets with sufficient amount of samples are very important for training deep convolutional neural networks (CNNs). However, many of the available real-world data sets contain erroneously labeled samples and those errors substantially hinder the learning of very accurate CNN models. In this work, we consider the problem of training a deep CNN model for image classification with mislabeled training samples - an issue that is common in real image data sets with tags supplied by amateur users. To solve this problem, we propose an auxiliary image regularization technique, optimized by the stochastic Alternating Direction Method of Multipliers (ADMM) algorithm, that automatically exploits the mutual context information among training images and encourages the model to select reliable images to robustify the learning process. Comprehensive experiments on benchmark data sets clearly demonstrate our proposed regularized CNN model is resistant to label noise in training data.
BibTeX
@inproceedings{feng2016auxiliary,
  author = {Jiashi Feng and Stefanie Jegelka and Trevor Darrell},
  title = {{Auxiliary Image Regularization for Deep CNNs with Noisy Labels Samaneh Azadi}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1511.07069},
}
Figure