A Discriminative Latent Model of Image Region and Object Tag Correspondence

Yang Wang, Greg MoriNeurIPS 2010 · OpenReview

Introduction

In computer vision, "image annotation" typically involves assigning a list of tags to an entire image. However, most traditional methods rely on a "bag-of-words" approach, treating the image as an unstructured collection of features and the tags as a simple list. This ignores a fundamental truth: specific words refer to specific parts of the image.

The primary challenge is that training data rarely comes with "aligned" labels. We might know an image contains a "horse" and a "rider," but we aren't explicitly told which pixels belong to which object. Because manually drawing bounding boxes around every object is prohibitively expensive, there is a strong need to learn these correspondences automatically from "unaligned" data.

The model learns to map unaligned image-level tags (a) to specific localized regions (c).

To address this, this paper introduces a discriminative latent model that automatically learns to associate specific image regions with textual tags (like "athlete" or "tree") without requiring manual alignment during training. By treating the region-to-tag mapping and the overall scene category as latent (hidden) variables within a Support Vector Machine (SVM) framework, the model simultaneously improves image annotation and performs unsupervised scene clustering.

Methodology

The authors propose a Discriminative Latent Model built upon the Latent SVM framework. Unlike generative models that attempt to model the entire distribution of images and words, this model uses a max-margin approach to focus specifically on the boundaries that separate correct annotations from incorrect ones.

The model incorporates two key hidden layers (latent variables):

  1. Mapping ($z$): A many-to-one assignment where each image region is mapped to at most one annotation term. Crucially, the model enforces a structural constraint: every active tag must be supported by at least one region.
  2. Scene Label ($s$): A hidden category (e.g., "polo match" or "sailing") that helps group similar images and provides context for which tags are likely to co-occur.
Illustration of the latent mapping variable, assigning image regions to active annotation tags.

To process the images, they are first segmented into regions. For each region, the model extracts four feature types—shape, texture, color, and location—which are then quantized into "visual words." Because finding the optimal tags for a new image is an NP-hard problem, the authors utilize Linear Programming (LP) relaxation to transform it into a solvable optimization problem, allowing the model to efficiently predict tags and their corresponding regions.

Experiments and Results

The model was evaluated on the UIUC Sport Dataset, which contains images from eight sports categories (badminton, bocce, croquet, polo, rock climbing, rowing, sailing, and snowboarding). Each image is paired with unaligned object-level tags (e.g., "water," "grass," "skier").

The proposed model outperformed standard baselines in two main areas:

  • Annotation Accuracy: Using the F-measure, the model achieved 0.4552, significantly beating a standard SVM baseline (0.4112) and a previous state-of-the-art topic model (0.3500).
  • Scene Clustering: The model’s ability to group images into scenes was measured using Normalized Mutual Information (NMI). It achieved a score of 0.5295, outperforming K-means (0.2227) and other sophisticated clustering baselines.
  • Qualitative Localization: The model successfully "localized" objects, overlaying colors on image regions that corresponded to specific tags (e.g., correctly identifying the "athlete" vs. the "floor" in a gym, as seen in the introductory figure).

Key Insights and Future Work

The success of the model highlights several important insights. First, context matters: by treating the "scene" as a latent variable, the model learns that certain objects are more compatible with certain environments (e.g., "water" is highly compatible with a "sailing" scene but not "rock climbing"). Second, the model naturally learns spatial priors, discovering on its own that "sky" usually appears at the top of an image, while "water" or "grass" appears at the bottom. Finally, by enforcing that every tag mentioned must be linked to at least one region, the model avoids "hallucinating" objects and produces a rough segmentation of the image as a useful by-product.

Despite these successes, the authors note a few limitations for future work. The current model assumes a region can only belong to one tag (a many-to-one assumption), whereas in reality, a single region might be described by multiple tags (e.g., "red" and "shirt"). Additionally, the vocabulary is currently constrained to nouns. Future iterations could extend this to verbs and adjectives, incorporate linguistic relationships (like "rider" usually being "on" a "horse"), and test the model on noisier real-world data like raw Flickr or YouTube tags.

Summary updated

Abstract

We propose a discriminative latent model for annotating images with unaligned object-level textual annotations. Instead of using the bag-of-words image representation currently popular in the computer vision community, our model explicitly captures more intricate relationships underlying visual and textual information. In particular, we model the mapping that translates image regions to annotations. This mapping allows us to relate image regions to their corresponding annotation terms. We also model the overall scene label as latent information. This allows us to cluster test images. Our training data consist of images and their associated annotations. But we do not have access to the ground-truth region-to-annotation mapping or the overall scene label. We develop a novel variant of the latent SVM framework to model them as latent variables. Our experimental results demonstrate the effectiveness of the proposed model compared with other baseline methods.
BibTeX
@inproceedings{wang2010discriminative,
  author = {Yang Wang and Greg Mori},
  title = {{A Discriminative Latent Model of Image Region and Object Tag Correspondence}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=B1-_yFbuZB},
}
Figure