IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing

Wenjun Zhang, Liangxiao Jiang, Chaoqun Li

Executive Summary

IWBVT is a universal post-processing framework that shifts the focus of crowdsourcing from merely cleaning labels to maximizing the performance of the models trained on them. By combining a novel entropy-based instance weighting scheme with probabilistic loss regression, it mitigates the impact of "intractable" (ambiguous) examples and achieves a superior bias-variance trade-off, significantly boosting model accuracy across dozens of datasets.

The Challenge: Label Quality vs. Model Quality

In crowdsourcing, multiple non-experts are often asked to label the same data point to mitigate noise. Traditionally, researchers have focused on label quality: how accurately can we infer the "true" label from a pile of noisy worker votes?

However, high label quality doesn't always lead to high model quality (the accuracy of a classifier trained on those labels). The culprit lies in intractable instances. These are data points with ambiguous attributes that confuse both workers and models. While these instances might be few enough that they don't tank the overall label accuracy of a dataset, they wreak havoc on a model's ability to learn clean classification boundaries. Existing algorithms often ignore these "hard" cases, leading to models that generalize poorly.

Proposed Method: IWBVT

The proposed approach, IWBVT (Instance Weighting-based Bias-Variance Trade-off), acts as a wrapper that can be applied to any existing label integration or noise correction algorithm. It works in two main stages:

1. Smart Instance Weighting

The authors introduce a new weighting metric based on the "complementary set" and entropy. The intuition is clever: for a "tractable" (clear) instance, workers should reach a consensus on one class, and their remaining votes should be randomly distributed (resulting in high entropy) across the other classes. Conversely, if the votes are concentrated on a few confusing classes, the instance is likely intractable and is assigned a lower weight during training.

Illustration of the instance weighting method based on the complementary set and its entropy.

2. Probabilistic Loss Regression

To further reduce generalization error, IWBVT performs a bias-variance trade-off. It trains a regression model to predict the "probabilistic loss"—the difference between the model's prediction and the integrated label. During inference, the model doesn't just use the classifier's output; it adjusts the prediction using the output of this regression model. Mathematically, this asymptotically minimizes the combination of bias and variance.

Key Insights and Ideas

  • Label Quality $\neq$ Model Quality: This is the core insight. You can have a dataset that is 95% "correctly" labeled, but if the remaining 5% of errors are concentrated on the most ambiguous parts of the feature space, your model's decision boundary will be significantly distorted.
  • The Power of the "Complementary Set": Most weighting schemes look only at the winning class. By looking at the entropy of the losing classes (the complementary set), IWBVT gains a much finer-grained understanding of how "confused" the crowd (and thus the model) is about a specific point.
Comparison of different weighting metrics on complex label distributions.
  • Universal Compatibility: IWBVT isn't a replacement for algorithms like Majority Voting (MV) or Expectation-Maximization (EM); it’s an upgrade. Even the simplest algorithm combined with IWBVT can outperform complex state-of-the-art methods.

Experimental Evaluation

The authors conducted an extensive evaluation using:

  • 34 Simulated Datasets: Sourced from the CEKA platform, covering diverse domains. Noise was simulated by generating five virtual workers with varying label qualities.
  • 2 Real-World Crowdsourced Datasets:
    • Leaves: 384 instances, 6 classes, annotated by 83 workers via Amazon Mechanical Turk (AMT).
  • Income: 600 instances, 2 classes, annotated by 67 workers via AMT.

Key Results

The experimental results are remarkably consistent:

  • Simulated Data: On average, across 34 datasets, IWBVT improved the model quality of Majority Voting from 75.45% to 79.03%.
  • Beating the SOTA: Majority Voting + IWBVT actually outperformed several standalone state-of-the-art noise correction algorithms (like AVNC and MVNC) in their original forms.
  • Real-World Impact: On the Leaves dataset, applying IWBVT to the MNLDP algorithm boosted model accuracy from roughly 55% to over 61%.
  • Statistical Significance: In head-to-head comparisons, IWBVT showed statistically significant improvements in the vast majority of cases, rarely degrading performance.

Limitations and Future Work

The authors identify one primary weakness: robustness to anomalous attributes. IWBVT currently uses linear regression for its probabilistic loss prediction. In datasets where certain numerical features have massive magnitudes (outliers) compared to others, the linear regression model struggles, which can degrade the final model quality.

Future Direction: The authors plan to explore more robust, non-linear regression techniques to handle these anomalous attributes, potentially making the framework even more universally applicable across messy, real-world data.

Summary updated

Abstract

In recent years, a large number of algorithms for label integration and noise correction have been proposed to infer the unknown true labels of instances in crowdsourcing. They have made great advances in improving the label quality of crowdsourced datasets. However, due to the presence of intractable instances, these algorithms are usually not as significant in improving the model quality as they are in improving the label quality. To improve the model quality, this paper proposes an instance weighting-based bias-variance trade-off (IWBVT) approach. IWBVT at first proposes a novel instance weighting method based on the complementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Then, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-variance trade-off and thus reduces the generalization error of trained models. Experimental results indicate that IWBVT can serve as a universal post-processing approach to significantly improving the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.
BibTeX
@inproceedings{zhang2024iwbvt,
  author = {Wenjun Zhang and Liangxiao Jiang and Chaoqun Li},
  title = {{IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=aJDGfynRw7},
}
Figure