Actively selecting annotations among objects and attributes

Adriana Kovashka, Sudheendra Vijayanarasimhan, Kristen GraumanICCV 2011 · OpenReview

Abstract

We present an active learning approach to choose image annotation requests among both object category labels and the objects' attribute labels. The goal is to solicit those labels that will best use human effort when training a multi-class object recognition model. In contrast to previous work in active visual category learning, our approach directly exploits the dependencies between human-nameable visual attributes and the objects they describe, shifting its requests in either label space accordingly. We adopt a discriminative latent model that captures object-attribute and attribute-attribute relationships, and then define a suitable entropy reduction selection criterion to predict the influence a new label might have throughout those connections. On three challenging datasets, we demonstrate that the method can more successfully accelerate object learning relative to both passive learning and traditional active learning approaches.

Introduction

Traditional active learning for object recognition typically operates in a one-dimensional space: an algorithm selects unlabeled images and requests category labels (e.g., "Is this a panda?") for the most informative instances. However, as the number of object categories scales into the thousands, this approach becomes increasingly expensive and inefficient.

This research introduces a multi-task active learning framework that allows a learner to choose between asking for an object category label or a descriptive attribute label (e.g., "Does this have spots?"). By exploiting the shared nature of visual attributes—such as "furry," "metallic," or "cylindrical"—across different object classes, the system significantly reduces the human effort required to train accurate multi-class recognition models.

Methodology

The proposed approach rests on two primary pillars: a unified model of objects and attributes and a selection criterion based on expected entropy reduction.

The Object-Attribute Model

The authors utilize a discriminative latent model (specifically a Latent SVM) to capture four critical types of relationships:

  1. Object Class: The direct visual appearance of the object.
  2. Attribute Class: The direct visual appearance of specific attributes.
  3. Attribute-Attribute Relations: Correlations between traits (e.g., "has skin" and "is metallic" are unlikely to co-occur).
  4. Object-Attribute Relations: The frequency with which a specific object class possesses a specific attribute.
The active learning framework chooses between object-level and attribute-level queries, leveraging a model that captures relationships between objects and their shared attributes.

Expected Entropy Reduction (EER)

To determine the most informative query, the system simulates the impact of every possible answer to every possible question. It calculates the Total Entropy—a measure of uncertainty—across all images in the dataset. It then selects the image-label pair expected to yield the greatest reduction in total entropy. Unlike heuristic methods, this approach uses a single mathematical objective to rank object and attribute queries side-by-side.

Key Insights

  • Attributes as Force Multipliers: Attribute labels are often more informative than object labels because they propagate information across the category hierarchy. Learning about "spots" helps distinguish leopards from lions and dalmatians from poodles simultaneously.
  • Preference for Attributes: In qualitative evaluations, the system chose attribute queries approximately 75% of the time, suggesting that describing visual traits is often more useful to a learner than simple categorization.
  • Global Impact: While traditional active learning focuses on local uncertainty (a single image), this method evaluates how a label helps the entire model across all images, which is essential for exploiting shared attributes.

Experimental Evaluation

The method was evaluated on three benchmark datasets: Animals with Attributes (AwA) (50 categories, 85 attributes), a-Pascal (20 classes, 64 attributes), and a-Yahoo (12 classes, 64 attributes).

Results

The proposed method consistently outperformed both passive (random) learning and standard active learning that only requested object labels. On the AwA dataset, the system reached 74.48% of its maximum potential confidence using only 4% of the total available labels. Standard object-only active learning sometimes performed worse than random sampling due to its inability to leverage the "knowledge transfer" provided by attributes. Furthermore, the entropy curves demonstrated that this approach reduces classification uncertainty much faster than baselines, closely tracking an "optimal" oracle.

Conclusion and Future Work

While highly effective, the selection process is computationally intensive, with costs growing quadratically relative to the number of images. Future research may address this bottleneck or expand the model to include relative attributes (e.g., "A is furrier than B") and direct queries about object-attribute relationships (e.g., "Do all tigers have stripes?") to further refine the model’s internal logic.

Summary updated

Abstract

We present an active learning approach to choose image annotation requests among both object category labels and the objects' attribute labels. The goal is to solicit those labels that will best use human effort when training a multi-class object recognition model. In contrast to previous work in active visual category learning, our approach directly exploits the dependencies between human-nameable visual attributes and the objects they describe, shifting its requests in either label space accordingly. We adopt a discriminative latent model that captures object-attribute and attribute-attribute relationships, and then define a suitable entropy reduction selection criterion to predict the influence a new label might have throughout those connections. On three challenging datasets, we demonstrate that the method can more successfully accelerate object learning relative to both passive learning and traditional active learning approaches.
BibTeX
@inproceedings{kovashka2011actively,
  author = {Adriana Kovashka and Sudheendra Vijayanarasimhan and Kristen Grauman},
  title = {{Actively selecting annotations among objects and attributes}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2011},
  url = {https://openreview.net/forum?id=BJ4El-zuZH},
}
Figure