TL;DR
This paper argues that the inherent variability in human labeling (noise) fundamentally limits how predictable supervised learning models can be, even when they achieve high accuracy. The authors demonstrate this by showing that models can sometimes predict human choices better than humans predict their own past choices, suggesting a "negative performance gap" that arises from ignoring labeling noise.
What Problem Does This Paper Address?
Supervised learning models are trained to predict outcomes based on labeled data, with the assumption that these labels represent a definitive "ground truth." However, in many real-world domains, human annotators disagree with each other, and even with themselves over time, when assigning labels. This is particularly true in subjective tasks like identifying toxic content or diagnosing medical conditions from images. The paper questions how we can interpret state-of-the-art models that achieve high accuracy in these noisy domains, especially when their predictive power exceeds human consistency. The core problem is understanding and quantifying the impact of this "imprecise oracle" (human labeler) on the limits of supervised learning predictability.
Methods
The authors propose a framework for understanding the impact of labeling noise using confusion matrices and metrics like F1-score and Cohen's kappa (κ). They derive a relationship showing that as labeling noise (measured by κ) increases, the achievable F1-score is bounded from above by 1.
To empirically investigate this, they conducted behavioral choice experiments. Participants were presented with risky choice problems multiple times over several weeks. This "test-retest" design allowed them to measure:
- Human-human consistency: How often different people agree on the same choice.
- Human-model consistency: How well existing choice models predict human choices.
- Intra-rater reliability: How consistent an individual is with their own past choices (measured using Cohen's κ).
They specifically looked at "Decisions from Description" (where choices are based on explicit descriptions of probabilities and payoffs) and "Decisions from Experience" (where choices are learned through trial and error).
Datasets
The paper utilizes data from several sources:
- RR (Repeated Risk): Data collected from their own experiments, involving repeated presentations of choice problems to participants over three weeks.
- TE (Technion Estimation) and TC (Technion Competition): Datasets from the first Technion Choice Prediction Tournament.
- EEP: Data from a mixed paradigm used in Plonsky et al. [Erev et al., 2017].
They use both "Decisions from Description" (DFD) and "Decisions from Experience" (DFE) paradigms from these sources.
Key Insights and Ideas
- Labeling noise is a fundamental limit: The paper mathematically demonstrates that the inherent variability in human labels (represented by Cohen's κ) directly constrains the maximum achievable predictive performance (F1-score).
- "Negative Performance Gap": In domains with high labeling noise, models can sometimes achieve higher agreement with the noisy labels than humans achieve with themselves. This suggests the model might be "overfitting" to the noise rather than capturing true underlying patterns.
- Choice models can be too consistent: For risky choice behavior, which is inherently stochastic, models that perfectly predict human choices might actually be poor explanations because they fail to capture the natural variability in human decision-making.
- The problem is widespread: The authors argue this issue isn't confined to niche domains but is prevalent in areas like medical diagnosis and even judicial risk assessment, where subjective judgment plays a significant role.
Key Results
- Human-human consistency is lower than model-data consistency: In their DFD experiments, human-human correlations for predicting R-rates ranged from 0.67-0.78, with agreement proportions of 78%-85%. These are significantly lower than previously reported model-data comparisons, even for baseline models.
- Low intra-rater reliability: Individual consistency (intra-rater reliability) for risky choices was low, with Cohen's κ around 0.25, reflecting the stochastic nature of human decision-making.
- Models agree with noisy data more than humans agree with themselves: Baseline models showed higher agreement with the noisy human choices in their dataset (κ = 0.23) than humans showed with their own past choices.
- DFE results are more aligned with human consistency: In Decisions from Experience, human-human correlations and agreement proportions were closer to model-human measurements, suggesting that simpler models might be sufficient for these tasks.
Limitations and Future Work
- Overfitting to validation sets: The authors suggest that the "negative performance gap" observed in some tournament settings might be partly due to models being overfitted to validation sets during architecture design. They propose restricting access to validation sets during model development as a technical solution.
- Need for incentivizing replica data annotation: To better understand predictability limits, they advocate for mechanisms that encourage and promote diverse, replicated data annotation efforts, especially in domains lacking objective ground truth.
- Beyond averaging: While averaging annotator inputs can reduce measurement noise, it won't eliminate the "process noise" inherent in the labeling itself.
- Focus on interpretability: The paper strongly suggests that understanding these predictability limits should drive the discourse on model selection, favoring interpretable models when complex, uninterpretable models claim predictive abilities that exceed reasonable bounds.
- Consequences of ignoring noise: The authors warn that in high-stakes applications, ignoring the stochasticity of data labels can lead to a false sense of competence in AI systems, with potentially serious consequences.
Summary updated