Annotator rationales for visual recognition

Jeff Donahue, Kristen GraumanICCV 2011 · OpenReview

Abstract

Traditional supervised visual learning simply asks annotators “what” label an image should have. We propose an approach for image classification problems requiring subjective judgment that also asks “why”, and uses that information to enrich the learned model. We develop two forms of visual annotator rationales: in the first, the annotator highlights the spatial region of interest he found most influential to the label selected, and in the second, he comments on the visual attributes that were most important. For either case, we show how to map the response to synthetic contrast examples, and then exploit an existing large-margin learning technique to refine the decision boundary accordingly. Results on multiple scene categorization and human attractiveness tasks show the promise of our approach, which can more accurately learn complex categories with the explanations behind the label choices.

Introduction

In the standard supervised learning paradigm, human annotators act as simple labeling oracles. They are presented with an image and asked to provide a categorical label—the "what." While effective for many tasks, this approach provides a narrow information channel, especially for complex or subjective visual categories. For instance, distinguishing between a "lounge" and a "bedroom" or judging human attractiveness involves nuanced decisions that a single label cannot fully capture.

This paper introduces the concept of Annotator Rationales. Instead of just asking for a label, we ask annotators why they chose that label. By capturing these underlying reasons—either as spatial regions or semantic attributes—we can significantly enrich the training signal and improve model performance, particularly when data is scarce.

Methodology: Learning from "Why"

The core of our approach lies in converting human explanations into a format that a machine learning model can ingest. We achieve this through the creation of Contrast Examples.

Spatial and Attribute Rationales

We explore two primary modes of rationales:

  1. Spatial Rationales: Annotators draw polygons around the specific image regions that most influenced their decision. For example, in a sports classification task, an annotator might highlight the specific pose or equipment of an athlete.
  2. Attribute Rationales: Annotators select from a predefined vocabulary of semantic properties (e.g., "smiling," "square face") that drove their judgment.
A figure skater's pose serves as a spatial rationale, highlighting the discriminative features that lead to a specific classification.

Generating Contrast Examples

For every training image $x_i$, we generate a synthetic "contrast example" $v_i$. For spatial rationales, $v_i$ is created by removing the features within the human-specified polygon in the descriptor space. For attribute rationales, $v_i$ is formed by artificially weakening the scores of the relevant attribute detectors.

The Secondary Margin Objective

We integrate these rationales into the learning process by modifying the standard Support Vector Machine (SVM) objective. We introduce a secondary margin ($\mu$) constraint: $$f(x_i) - f(v_i) \geq \mu$$ This constraint forces the classifier to assign a significantly higher confidence score to the original image $x_i$ than to its contrast version $v_i$. Effectively, the model learns that if the features identified by the human are removed or weakened, the evidence for the category should diminish.

Experimental Evaluation

We evaluated our method across three datasets focusing on scene recognition and human attractiveness.

Scene Categorization

Using the 15 Scene Categories benchmark, we collected over 8,000 spatial rationales via Amazon Mechanical Turk. The rationale-augmented model improved Mean Average Precision (MAP) in 13 out of 15 categories. The gains were most pronounced in categories defined by specific objects, such as "Kitchen" or "Store," where rationales helped the model focus on discriminative items like stoves or cash registers.

Scene recognition tasks, such as identifying a bedroom, benefit from rationales that pinpoint defining objects like the bed or dresser.

Human Attractiveness

We introduced a "Hot or Not" dataset and used the PubFig database to test both spatial and attribute rationales. In the attractiveness task, human rationales often pointed to non-facial cues like clothing or posture. Our results showed that models trained with these human-selected rationales outperformed those using simple face-cropping baselines, demonstrating that the "why" behind human judgment captures essential context that automated crops miss.

Key Insights

  • Rationales vs. Segmentation: Unlike full object segmentation, rationales focus strictly on discriminative parts. In a "store" scene, the cash register is the rationale, even if the floor is also part of the foreground.
  • Data Efficiency: Rationales act as a high-density information source. We found that a model trained on just 25 images with rationales could outperform a model trained on 100 images with labels only.
  • Amortized Effort: Collecting rationales adds minimal overhead to the annotation process but provides a substantial boost in learning signal.

Conclusion

By moving beyond simple labels to incorporate annotator rationales, we can build models that better reflect the complexity of human visual judgment. This approach not only improves accuracy in subjective tasks but also offers a path toward more data-efficient supervised learning. Future work will explore combining spatial and semantic rationales into a unified framework and applying these techniques in active learning scenarios.

Summary updated

Abstract

Traditional supervised visual learning simply asks annotators “what” label an image should have. We propose an approach for image classification problems requiring subjective judgment that also asks “why”, and uses that information to enrich the learned model. We develop two forms of visual annotator rationales: in the first, the annotator highlights the spatial region of interest he found most influential to the label selected, and in the second, he comments on the visual attributes that were most important. For either case, we show how to map the response to synthetic contrast examples, and then exploit an existing large-margin learning technique to refine the decision boundary accordingly. Results on multiple scene categorization and human attractiveness tasks show the promise of our approach, which can more accurately learn complex categories with the explanations behind the label choices.
BibTeX
@inproceedings{donahue2011annotator,
  author = {Jeff Donahue and Kristen Grauman},
  title = {{Annotator rationales for visual recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2011},
  url = {https://openreview.net/forum?id=Sk-pYgzubH},
}
Figure