CS-Isolate: Extracting Hard Confident Examples by Content and Style Isolation

Yexiong Lin, Yu Yao, Xiaolong Shi, Mingming Gong, Xu Shen, Dong Xu, Tongliang Liu

TL;DR

CS-Isolate improves how machine learning models handle noisy labels by disentangling an image's "content" (the actual object) from its "style" (background, lighting, etc.). By isolating these factors, the method can identify "hard" but correctly labeled examples that traditional methods often mistake for noise, leading to significantly better generalization on real-world datasets.

The Problem: Label Noise and "Hard" Examples

Large-scale datasets are rarely perfect; they are often riddled with "label noise" (incorrectly labeled images). Deep neural networks are notorious for memorizing these errors, which hurts their ability to generalize to new data.

Current state-of-the-art methods use a "small-loss trick": they assume that examples the model learns easily (low loss) are likely clean, while high-loss examples are noisy. However, this creates a blind spot for hard confident examples. These are images that are correctly labeled but sit near the decision boundary—perhaps because the object is partially obscured or the lighting is weird. Because these "hard" examples have high loss, current models often discard them as noise.

Conceptual illustration of content and style isolation

The authors argue that these examples are "hard" primarily because their style factors (unhelpful information) are entangled with their content factors (semantic information).

The CS-Isolate Method

The authors introduce CS-Isolate (Content and Style Isolation), a framework that can be plugged into existing noise-robust algorithms like DivideMix or Co-Teaching.

Key components include:

  • Identifiable Variational Autoencoder (iVAE): The core architecture used to infer latent factors. Unlike standard VAEs, iVAEs use auxiliary variables to ensure the learned representations are "identifiable" (meaning the model actually recovers the true underlying factors).
  • Style Surrogates via Augmentation: Since we don't know the "true" style of an image, the authors use data augmentation (rotation, color jitter, etc.) as a proxy. They assign a unique "style ID" to each type of augmentation, forcing the model to learn what changes (style) versus what stays the same (content).
  • Dynamic Content ID Refinement: Initially, every image is treated as having unique content. As the model identifies "confident" examples during training, it reassigns them a shared "content ID" based on their predicted class. This encourages the model to learn a consistent content representation for all images in the same category.
  • End-to-End Training: The model simultaneously minimizes a reconstruction loss (to learn style/content) and a classification loss (to predict labels), balanced using a Lagrangian approach.

Key Insights

  • Entanglement is the Root of "Hardness": A "horse" image might be hard to classify not because the horse is blurry, but because there is a prominent "person" in the background. Traditional models get distracted by the person (style); CS-Isolate learns to ignore the person and focus on the horse (content).
  • Augmentation as Intervention: By treating data augmentation as a controlled change in style, the authors turn a standard pre-processing step into a powerful tool for representation learning.

Datasets and Key Results

The researchers evaluated CS-Isolate on a mix of synthetic and real-world noisy benchmarks, including Fashion-MNIST, CIFAR-10, CIFAR-100 (with instance-dependent noise), CIFAR-10N, and Clothing1M.

Test accuracy on Fashion-MNIST, CIFAR-10, and CIFAR-100 with instance-dependent noise

CS-Isolate consistently outperformed existing methods across the board:

  • Better Selection: On the "worst" noise setting of CIFAR-10N, CS-Isolate achieved a recall of 98.0% for confident examples, compared to 96.6% for DivideMix. This means it successfully "saved" many hard examples that other models threw away.
  • Higher Accuracy: On CIFAR-100 with 40% instance-dependent noise (IDN-0.4), CS-Isolate reached 73.11% accuracy, a significant jump over DivideMix’s 70.50%.
  • Real-World Performance: On the challenging Clothing1M dataset, it achieved a top-tier accuracy of 74.92%.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Uncontrolled Styles: While data augmentation helps, it cannot control all style factors. For example, if a "person" always appears in "horse" photos in the training set, the model might still struggle to separate them without more diverse data.
  • Computational Overhead: Because the method involves a generative component (the VAE) and multiple augmentations per image, it is computationally more expensive than simple classification.
  • Future Directions: The authors suggest exploring more sophisticated ways to define "style" beyond simple augmentations and applying this disentanglement logic to other domains like natural language processing.

Summary updated

Abstract

Label noise widely exists in large-scale image datasets. To mitigate the side effects of label noise, state-of-the-art methods focus on selecting confident examples by leveraging semi-supervised learning. Existing research shows that the ability to extract hard confident examples, which are close to the decision boundary, significantly influences the generalization ability of the learned classifier. In this paper, we find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. The hard examples become more discriminative when we focus solely on content factors, such as semantic information, while ignoring style factors. Nonetheless, given only noisy data, content factors are not directly observed and have to be inferred. To tackle the problem of inferring content factors for classification when learning with noisy labels, our objective is to ensure that the content factors of all examples in the same underlying clean class remain unchanged as their style information changes. To achieve this, we utilize different data augmentation techniques to alter the styles while regularizing content factors based on some confident examples. By training existing methods with our inferred content factors, CS-Isolate proves their effectiveness in learning hard examples on benchmark datasets. The implementation is available at https://github.com/tmllab/2023_NeurIPS_CS-isolate.
BibTeX
@inproceedings{lin2023cs,
  author = {Yexiong Lin and Yu Yao and Xiaolong Shi and Mingming Gong and Xu Shen and Dong Xu and Tongliang Liu},
  title = {{CS-Isolate: Extracting Hard Confident Examples by Content and Style Isolation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=Lkc0KjsDFv},
}
Figure