IDE: A System for Iterative Mislabel Detection

Yuhao Deng, Qiyan Deng, Chengliang Chai, Lei Cao, Nan Tang, Ju Fan, Jiayi Wang, Ye Yuan, Guoren Wang

Abstract

While machine learning techniques, especially deep neural networks, have shown remarkable success in various applications, their performance is adversely affected by label errors in training data. Acquiring high-quality annotated data is both costly and time-consuming in real-world scenarios, requiring extensive human annotation and verification. Consequently, many industry-applied models are trained over data containing substantial noise, significantly degrading the performance of these models.To address this critical issue, we demonstrate IDE, a novel system that iteratively detects mislabeled instances and repairs the wrong labels. Specifically, IDE leverages the early loss observation and influence-based verification to iteratively identify mislabeled instances. When the mislabeled instances are obtained in each iteration, IDE will repair their labels to enhance detection accuracy for subsequent iterations. The framework automatically determines the termination point when the early loss is no longer effective. For uncertain instances, it generates pseudo labels to train a binary classification model, leveraging the model's generalization ability to make the final decision. With a real-life scenario, we demonstrate that IDE produces high-quality training data by effective mislabel detection and repair.
BibTeX
@inproceedings{deng2024ide,
  author = {Yuhao Deng and Qiyan Deng and Chengliang Chai and Lei Cao and Nan Tang and Ju Fan and Jiayi Wang and Ye Yuan and Guoren Wang},
  title = {{IDE: A System for Iterative Mislabel Detection}},
  booktitle = {SIGMOD Conference Companion},
  year = {2024},
  doi = {10.1145/3626246.3654737},
}

Loading related content...

Figure