Abstract
Imagine we show an image to a person and ask her/him to decide whether the scene in the image is warm or not warm, and whether it is easy or not to spot a squirrel in the image. For exactly the same image, the answers to those questions are likely to differ from person to person. This is because the task is inherently ambiguous. Such an ambiguous, therefore challenging, task is pushing the boundary of computer vision in showing what can and can not be learned from visual data. Crowdsourcing has been invaluable for collecting annotations. This is particularly so for a task that goes beyond a clear-cut dichotomy as multiple human judgments per image are needed to reach a consensus. This paper makes conceptual and technical contributions. On the conceptual side, we define disagreements among annotators as privileged information about the data instance. On the technical side, we propose a framework to incorporate annotation disagreements into the classifiers. The proposed framework is simple, relatively fast, and outperforms classifiers that do not take into account the disagreements, especially if tested on high confidence annotations.
Introduction
In computer vision, crowdsourcing platforms like Amazon Mechanical Turk are essential for labeling large-scale datasets. However, many visual tasks are inherently ambiguous—for instance, determining if a scene is "warm" or if an object is "easy to spot." Standard practice typically involves using majority voting to establish a "ground truth," effectively discarding the variance in human responses as noise.
This paper argues that such disagreements are not merely noise but represent privileged information about the data instance. By treating annotator uncertainty as a valuable signal rather than a nuisance, the authors propose a framework to incorporate these disagreements into classifiers, demonstrating that "ambiguity helps" improve model robustness and accuracy.
Methodology: Learning Under Privileged Information
The proposed framework operates within the Learning Under Privileged Information (LUPI) paradigm. Privileged information is data available during training (e.g., the distribution of worker votes) but unavailable during inference.
The GPCconf Model
The core technical contribution is the GPCconf, a novel Gaussian Process Classifier. Unlike standard GPCs, it utilizes two latent functions:
- Function $f$: The primary classifier predicting labels from image features.
- Function $g$: A "difficulty" function learned from privileged confidence scores.
When function $g$ identifies an image as "difficult" (high disagreement), the model reduces the influence of that data point on classifier $f$ by pushing its likelihood toward 0.5 (maximum uncertainty). This mechanism allows the model to learn more effectively from clear examples while remaining robust to ambiguous ones.
To ensure computational feasibility, the authors developed a quadrature-free Expectation Propagation (EP) algorithm. This makes GPCconf significantly faster than previous heteroscedastic models like GPC+ without sacrificing predictive accuracy.
Experimental Evaluation
The approach was evaluated on two primary benchmarks:
- SUNAttribute: 14,340 scene images with 102 attributes. Confidence scores were derived from the percentage of worker agreement on an attribute's presence.
- Animals with Attributes (AwA): Images ranked by humans from "easiest" to "hardest" to spot, converted into confidence scores for binary classification.
Features were extracted using 4096-dimensional deep CNN features (the fc7 layer of CaffeNet) pretrained on the Places and ImageNet databases.
Key Results and Discussion
The results demonstrate that accounting for ambiguity consistently improves performance. On the SUNAttribute dataset, GPC-based LUPI methods outperformed standard baselines in 58 out of 83 categories.
Efficiency and Accuracy
GPCconf proved to be highly efficient, training in 32 minutes on SUNAttribute tasks compared to 51 minutes for GPC+ and over 100 minutes for SVM+. Furthermore, the model showed significant gains when tested on high-confidence annotations, suggesting that learning from ambiguity during training helps the model better identify "obvious" cases during deployment.
Conclusion
This research highlights that human disagreement is a form of metadata that can be leveraged to build better machine learning models. While the current implementation focuses on binary classification and requires multiple annotations per training sample, the authors suggest that a promising future direction is integrating this ambiguity-aware loss directly into the training of convolutional layers.
Summary updated