IOFM: Using the Interpolation Technique on the Over-Fitted Models to Identify Clean-Annotated Samples

Dongha Kim, Yongchan Choi, Kunwoong Kim, Ilsang Ohn, Yongdai KimAAAI 2024 · DOI

TL;DR

This paper introduces IOFM, a novel method for identifying clean data in datasets with noisy labels. Unlike existing methods that rely on the "memorization effect," IOFM leverages the behavior of over-fitted deep neural networks to detect discrepancies between feature space and input space similarities, offering improved robustness, especially in challenging scenarios like imbalanced or heavily noisy datasets.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) often requires massive amounts of labeled data. While collecting large datasets is relatively easy, ensuring the accuracy of these labels is a significant challenge. Manual annotation is expensive and time-consuming, leading to datasets with "noisy labels" – incorrect annotations. This noisy label problem hinders the successful training of DNNs. Existing methods often exploit the "memorization effect," where DNNs learn clean data before noisy data. However, this effect can diminish in scenarios with imbalanced class distributions or high noise levels, leading to suboptimal performance. This paper addresses this limitation by proposing a new approach that doesn't solely rely on the memorization effect and performs well even when it's weak.

Methods

The core of the proposed method, IOFM (Interpolation with the Over-Fitted Model), is based on a novel observation: over-fitted DNNs exhibit different behaviors when interpolating between a data point and its nearest neighbor in the feature space, depending on whether the data point's label is clean or noisy.

Specifically, IOFM works by:

  • Over-fitting a DNN: The model is trained until it starts to over-fit the training data.
  • Neighborhood Analysis: For a given training sample $(x^*, y^*)$, its nearest neighbor in the feature space, $x_{nbd}$, is identified.
  • Interpolation and Score Calculation: The method then analyzes the DNN's prediction probability ($p_y(x)$) along the line segment connecting $x^*$ and $x_{nbd}$ in the input space.
  • If $y^*$ is clean, $x^*$ and $x_{nbd}$ are likely to be in similar regions of the input space, and $p_y(x)$ remains high along the interpolation. This results in a large IOFM score.
  • If $y^*$ is noisy, $x^*$ and $x_{nbd}$ might be distant in the input space, and $p_y(x)$ can drop significantly along the interpolation. This results in a small IOFM score.
  • Ensemble and MixUp: To further improve robustness, IOFM can use multiple scores from different training epochs (similar to AUM) and incorporates the MixUp loss function during training to enhance model smoothness.

A key advantage is that IOFM does not require fine-tuning the optimal training epoch, a common sensitivity in memorization-effect-based methods.

Datasets

The paper evaluates IOFM on several standard datasets:

  • CIFAR10 and CIFAR100: Small image datasets used for evaluating performance under various noise types (symmetric, asymmetric, instance-dependent) and imbalance strategies (Step, Long-tail).
  • Mini-ImageNet and Clothing1M: Larger image datasets used to demonstrate the method's scalability.

The authors also mention using a "two-moon" dataset for illustrative purposes in their theoretical analysis and visualizations.

Key Insights and Ideas

  • Feature Space vs. Input Space Discrepancy: The most crucial insight is that the relationship between a data point and its feature-space neighbors behaves differently in the original input space when the label is noisy versus clean. This provides a new angle for noisy label detection beyond just memorization.
  • Over-fitting as a Feature: Instead of avoiding over-fitting, the paper leverages it. The over-fitted state of a DNN is precisely what reveals the subtle differences in interpolation behavior that IOFM exploits.
  • Robustness to Imbalance and High Noise: IOFM's ability to perform well even when the memorization effect is weak is a significant advancement, opening doors for its application in more challenging, real-world scenarios.
  • Epoch-Agnostic Performance: The method's insensitivity to the exact training epoch is a major practical advantage, simplifying its application and reducing hyperparameter tuning effort.

Key Results

  • Superior Classification Accuracy: IOFM consistently outperforms existing state-of-the-art methods, including those based on the memorization effect, in identifying clean/noisy labels across various datasets and noise conditions. For instance, on heavily corrupted CIFAR10 (90% noise), IOFM enhances DivideMix by nearly 10%.
  • Competitive Performance with sinIOFM: A single-epoch version of IOFM (sinIOFM) is shown to be competitive with ensemble-based methods, offering a good trade-off between performance and computational cost.
  • Improved DivideMix: When combined with DivideMix, a popular semi-supervised learning framework for noisy labels, IOFM significantly boosts its performance, achieving the best or second-best accuracies in most tested scenarios.

Limitations and Future Work

  • Theoretical Guarantee vs. Practicality: While Theorem 1 guarantees the existence of a suitable DNN, the paper acknowledges that its practical obtainability is supported by empirical evidence and theoretical studies on over-parameterized DNNs.
  • Computational Cost: While IOFM is designed to be efficient, the paper mentions that neighborhood search can be computationally intensive for very large datasets. They propose a sampling strategy to mitigate this, which significantly reduces computation time without a substantial performance drop.
  • Future Directions: The authors suggest exploring IOFM's combination with other ME-based learning frameworks and its application to supervised anomaly detection tasks, where imbalanced and potentially noisy labels are common. They also highlight the potential for developing improved learning algorithms by further exploring the IOFM concept.

Summary updated

BibTeX
@inproceedings{kim2024iofm,
  author = {Dongha Kim and Yongchan Choi and Kunwoong Kim and Ilsang Ohn and Yongdai Kim},
  title = {{IOFM: Using the Interpolation Technique on the Over-Fitted Models to Identify Clean-Annotated Samples}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i12.29209},
}
Figure