Introduction
In supervised machine learning, high-quality labels are essential. However, human annotators often encounter ambiguous cases—such as a dog that closely resembles a wolf—leading to hesitation. To maintain data integrity, many annotation systems include an "unsure" option. Traditionally, these responses are discarded as noise, leading to significant data waste (often 10-30% of a dataset) and potential overfitting due to reduced training set sizes.
The LEUR Framework
The core contribution of this research is the LEUR (Learning with Unsure Responses) framework. The authors posit that "unsure" instances are not merely noise but provide a high-quality spatial signal: they tend to cluster near the ground-truth decision boundary.
The Unsure Loss Function
LEUR introduces a novel loss function ($H$) that acts as a regularizer. While standard loss functions penalize incorrect labels, the unsure loss penalizes the model for placing the decision boundary far from "unsure" instances. The researchers evaluated three variants:
- Squared Unsure Loss ($H_{sq}$): Penalizes the square of the distance to the boundary.
- Log Unsure Loss ($H_{log}$): Employs a logarithmic approach.
- Absolute Unsure Loss ($H_{abs}$): Penalizes the absolute distance.
The framework is model-agnostic and can be integrated into Logistic Regression, SVMs, or neural networks. It also extends to crowdsourcing via the LEUR-PC (Personal Classifier) model, which accounts for individual annotator biases while using unsure responses to anchor a global boundary.
Experimental Evaluation
The approach was tested on two datasets:
- MNIST (Synthetic): Digits "0" and "6", where uncertainty was simulated based on proximity to the boundary.
- DOG-WOLF (Real-world): A benchmark of 400 ImageNet images labeled by five human annotators, yielding a 15.6% unsure rate.
Key Results
The LEUR framework demonstrated significant advantages, particularly in low-data regimes. On the MNIST task, LEUR-PC outperformed standard baselines when provided with only 100 to 300 responses. In the real-world DOG-WOLF dataset, LEUR-PC variants achieved F1 scores of 0.63, a notable improvement over the popular Raykar method (0.51) and the standard Personal Classifier (0.60). Furthermore, the method proved robust across varying "unsure ratios," maintaining performance even when up to 40% of labels were "unsure."
Limitations and Future Work
While promising, the framework currently focuses on binary classification. Additionally, performance is sensitive to the hyperparameter $\gamma$, which balances the weight of the unsure loss against standard labels. Future work may extend these principles to multi-class problems and active learning scenarios, where models can strategically query for labels on the most ambiguous instances to further refine the decision boundary.
Summary updated