Learning with Unsure Responses

Kunihiro Takeoka, Yuyang Dong, Masafumi Oyamada

Introduction

In supervised machine learning, high-quality labels are essential. However, human annotators often encounter ambiguous cases—such as a dog that closely resembles a wolf—leading to hesitation. To maintain data integrity, many annotation systems include an "unsure" option. Traditionally, these responses are discarded as noise, leading to significant data waste (often 10-30% of a dataset) and potential overfitting due to reduced training set sizes.

Annotation interface with Dog, Wolf, and Unsure options.

The LEUR Framework

The core contribution of this research is the LEUR (Learning with Unsure Responses) framework. The authors posit that "unsure" instances are not merely noise but provide a high-quality spatial signal: they tend to cluster near the ground-truth decision boundary.

Unsure labels clustering near the decision boundary.

The Unsure Loss Function

LEUR introduces a novel loss function ($H$) that acts as a regularizer. While standard loss functions penalize incorrect labels, the unsure loss penalizes the model for placing the decision boundary far from "unsure" instances. The researchers evaluated three variants:

  • Squared Unsure Loss ($H_{sq}$): Penalizes the square of the distance to the boundary.
  • Log Unsure Loss ($H_{log}$): Employs a logarithmic approach.
  • Absolute Unsure Loss ($H_{abs}$): Penalizes the absolute distance.

The framework is model-agnostic and can be integrated into Logistic Regression, SVMs, or neural networks. It also extends to crowdsourcing via the LEUR-PC (Personal Classifier) model, which accounts for individual annotator biases while using unsure responses to anchor a global boundary.

Experimental Evaluation

The approach was tested on two datasets:

  1. MNIST (Synthetic): Digits "0" and "6", where uncertainty was simulated based on proximity to the boundary.
  2. DOG-WOLF (Real-world): A benchmark of 400 ImageNet images labeled by five human annotators, yielding a 15.6% unsure rate.

Key Results

The LEUR framework demonstrated significant advantages, particularly in low-data regimes. On the MNIST task, LEUR-PC outperformed standard baselines when provided with only 100 to 300 responses. In the real-world DOG-WOLF dataset, LEUR-PC variants achieved F1 scores of 0.63, a notable improvement over the popular Raykar method (0.51) and the standard Personal Classifier (0.60). Furthermore, the method proved robust across varying "unsure ratios," maintaining performance even when up to 40% of labels were "unsure."

Limitations and Future Work

While promising, the framework currently focuses on binary classification. Additionally, performance is sensitive to the hyperparameter $\gamma$, which balances the weight of the unsure loss against standard labels. Future work may extend these principles to multi-class problems and active learning scenarios, where models can strategically query for labels on the most ambiguous instances to further refine the decision boundary.

Summary updated

Abstract

Many annotation systems provide to add an unsure option in the labels, because the annotators have different expertise, and they may not have enough confidence to choose a label for some assigned instances. However, all the existing approaches only learn the labels with a clear class name and ignore the unsure responses. Due to the unsure response also account for a proportion of the dataset (e.g., about 10-30% in real datasets), existing approaches lead to high costs such as paying more money or taking more time to collect enough size of labeled data. Therefore, it is a significant issue to make use of these unsure.In this paper, we make the unsure responses contribute to training classifiers. We found a property that the instances corresponding to the unsure responses always appear close to the decision boundary of classification. We design a loss function called unsure loss based on this property. We extend the conventional methods for classification and learning from crowds with this unsure loss. Experimental results on realworld and synthetic data demonstrate the performance of our method and its superiority over baseline methods.
BibTeX
@inproceedings{takeoka2020learning,
  author = {Kunihiro Takeoka and Yuyang Dong and Masafumi Oyamada},
  title = {{Learning with Unsure Responses}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5355},
}
Figure