CrowdTeacher: Robust Co-teaching with Noisy Answers & Sample-specific Perturbations for Tabular Data

Mani Sotoodeh, Li Xiong, Joyce C. Ho

Abstract

Crowdsourcing is a cost-effective alternative to expert labeling, particularly in specialized domains like medicine. However, crowd-generated labels are inherently noisy and sparse. We present CrowdTeacher, a robust learning framework specifically designed for tabular data. CrowdTeacher integrates uncertainty-aware data perturbations with an asymmetric Co-teaching architecture. By modulating input feature perturbations based on the certainty of aggregated crowd labels, the framework prevents model over-memorization of noise. Experimental results on synthetic and real-world medical datasets demonstrate that CrowdTeacher significantly outperforms standard label-aggregation baselines and vanilla Co-teaching methods, especially in scenarios with high label sparsity.

Introduction

High-quality labeled data is a prerequisite for training effective machine learning models. In domains such as healthcare, obtaining expert annotations is often prohibitively expensive. Crowdsourcing offers a scalable alternative but introduces significant label noise due to varying annotator expertise. While truth inference algorithms like Dawid-Skene can estimate true labels, the resulting targets remain imperfect. Deep learning models tend to "memorize" this noise rather than learning underlying patterns. While Co-teaching—a technique where two neural networks filter out noisy samples for each other—has addressed label noise in computer vision, its application to tabular data remains under-explored. This paper introduces CrowdTeacher to address these challenges specifically for tabular datasets.

Methodology

CrowdTeacher employs three core components to enhance robustness. The mathematical notation used to define the framework components is detailed in Table 1.

Table 1: Symbols and descriptions used in the CrowdTeacher framework.

1. Sample-specific Perturbations

A truth inference model calculates a certainty score $c_i$ for each sample. Samples with high label disagreement (low certainty) undergo greater input feature perturbation. The more uncertain the label, the more the input is modified using synthetic data. This forces the model to be more robust to variations in samples where the label is "shaky."

2. Synthetic Data Generation

To create realistic perturbations, the framework utilizes tabular data synthesizers to model the underlying feature distributions. The authors evaluated three synthesizers: CTGAN, TVAE, and Gaussian Copula. Interestingly, the Gaussian Copula often outperformed more complex GAN-based models for generating the synthetic data used for perturbations in tabular settings.

3. Asymmetric Co-teaching

Standard Co-teaching uses two identical networks. However, tabular datasets are often smaller than image datasets, leading to high variance. CrowdTeacher uses a student-teacher architecture (one simple network, one complex) to improve convergence and prevent the networks from collapsing into the same biased state.

Experiments and Results

The framework was evaluated on synthetic datasets (40 features from various statistical families) and the Pressure Ulcer Injury (PUI) dataset from MIMIC-III, containing over 10,000 patient records. Performance was measured using AUPRC to account for class imbalance.

  • Superiority over Baselines: CrowdTeacher consistently outperformed "Vanilla" Co-teaching, Crowdlayer, and standard classifiers trained on aggregated labels.
  • Optimal Perturbation: An optimal perturbation fraction ($\alpha \approx 0.11$) was identified; too little perturbation doesn't help, while too much washes out the signal.
  • Robustness to Sparsity: The model remains robust even with an average of only ~1.7 labels per sample. Performance typically plateaus after 2 labels per sample, suggesting that expert-level results can be achieved without an excessive number of crowdworkers.

Conclusion

CrowdTeacher provides a robust solution for training on noisy, crowdsourced tabular data. By linking label uncertainty to feature-space perturbations, it effectively regularizes the learning process and prevents the memorization of noise. While performance depends on the quality of the data synthesizer, the framework offers a promising blueprint for leveraging crowd-labeled data in sensitive domains like healthcare. Future work will explore extensions to regression tasks and the integration of semi-supervised learning to further reduce reliance on expensive human annotations.

Summary updated

BibTeX
@inproceedings{sotoodeh2021crowdteacher,
  author = {Mani Sotoodeh and Li Xiong and Joyce C. Ho},
  title = {{CrowdTeacher: Robust Co-teaching with Noisy Answers \& Sample-specific Perturbations for Tabular Data}},
  booktitle = {PAKDD},
  year = {2021},
  url = {https://openreview.net/forum?id=YTBumhnLf1},
}
Figure