<title>Recognition as Translating Images into Text</title>

Kobus Barnard, Pınar Duygulu, David ForsythProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2003 · DOI

TL;DR

This paper introduces a novel paradigm for computer vision: treating image recognition as a translation problem from visual representations to semantic text. By leveraging large quantities of readily available image-text pairs (like web images with captions), they build statistical models to bridge this gap, enabling more nuanced understanding of image content and object recognition.

What Problem Does This Paper Address?

Traditional computer vision often struggles with understanding the semantic meaning of images, especially at a granular level. While object recognition has seen progress, answering questions like "what constitutes an object?" or "which objects are hard to distinguish?" remains challenging. The authors highlight the difficulty of obtaining large, precisely labeled datasets for supervised learning. They propose a solution by framing recognition as a translation task, mapping visual features to textual descriptions, and utilizing the abundance of naturally occurring image-text associations on the web and in other large collections. This approach aims to overcome the limitations of manual annotation and enable more flexible and robust image understanding.

Methods

The core of the approach is a "translation model" inspired by statistical machine translation. Images are first segmented into regions, and each region is represented by a set of features (size, position, color, texture, shape). The key innovation is learning a probabilistic mapping between these visual "blobs" and textual words.

Two main model families are explored:

  • Discrete Translation: This model uses K-means clustering to quantize image region features into "blob tokens." It then learns a joint probability distribution between these blob tokens and word tokens from associated text, treating it as a missing data problem solvable with Expectation-Maximization (EM).
  • Hierarchical Clustering Models: These models use a hierarchical structure (like a tree) to represent aspects of images and their associated text. They offer more sophisticated ways to model the joint probability of regions and words, allowing for integrated learning of feature characterizations and translation. Two variants are discussed: "independent" (where region features and words are emitted separately) and "dependent" (where words and blobs are emitted as pairs, requiring graph matching for correspondence).

The paper also discusses methods for evaluating these models, focusing on both "annotation performance" (predicting words for an entire image) and "correspondence performance" (linking specific words to image regions).

Datasets

The primary dataset used for experiments is the Corel image dataset, consisting of 160 CDs with approximately 100 images per CD on specific topics (e.g., "aircraft"). For experiments, a subset of 80 CDs was used for training and held-out data, while the remaining CDs formed a more challenging "novel" held-out set. The authors also mention using data from the Fine Arts Museum of San Francisco and news photos with captions as examples of readily available text-associated image collections.

Key Insights and Ideas

  • Vision as Translation: Reframing computer vision as a translation problem from visual to semantic representations is a powerful paradigm shift. It allows leveraging existing machine translation techniques and readily available data.
  • Leveraging Abundant "Noisy" Data: The approach thrives on large quantities of imperfectly associated text and images, rather than requiring meticulously labeled datasets. This makes it more scalable and practical.
  • Handling Ambiguity: The statistical models inherently handle ambiguity. For instance, a grey patch could be "pavement" or "mud," and the model learns to disambiguate based on context and co-occurrence statistics.
  • Understanding "What Counts as an Object": By treating every word as a potential object label, the models can implicitly learn what constitutes a recognizable object based on how reliably it can be associated with visual features.
  • Cluster Context Can Be Hindrance: Surprisingly, for novel images (significantly different from training data), using clustering context can actually hinder performance, suggesting that learned clusters don't generalize well to unseen domains.

Key Results

  • Annotation Performance: The "dependent-binary-tree" model achieved the best annotation performance on the held-out data (0.163), outperforming the "discrete-translation" model (0.122). However, for novel data, the dependent models performed worse than independent ones, with "discrete-translation" showing the least detriment (0.028).
  • Correspondence Performance: Table 2 shows that "discrete-translation" achieved a correspondence performance score of 0.91, significantly outperforming the other models (ranging from 0.96 to 0.100). This suggests that while hierarchical models might offer better annotation, the simpler discrete translation approach is more effective for learning direct region-word correspondences.
  • Empirical Distribution Baseline: The models consistently performed better than simply using the empirical word frequency distribution (around 0.19 for annotation proxy).

Limitations and Future Work

  • Correspondence Difficulty: Accurately measuring correspondence performance is challenging and requires manual labeling, which is time-consuming and prone to noise.
  • Cluster Context Generalization: The learned clusters from training data did not generalize well to novel images, indicating a need for more robust clustering or alternative context-modeling strategies.
  • Handling "NULL" Correspondence: The paper acknowledges the difficulty of handling cases where regions don't correspond to any word, or vice-versa, and suggests appending special "NULL" tokens as a potential solution, though fitting such models is complex.
  • Supervision Impact: The authors plan to investigate how small amounts of supervisory input can significantly improve model performance.
  • Vision Tool Evaluation: The developed testing paradigm can be used to evaluate other vision tools like segmentation algorithms and features.
  • Region Merging: The region word posteriors can be used to propose region merges, aiding in learning object shape and improving segmentation.

Summary updated

Abstract

We present an overview of a new paradigm for tackling long standing computer vision problems. Specifically our approach is to build statistical models which translate from a visual representations (images) to semantic ones (associated text). As providing optimal text for training is difficult at best, we propose working with whatever associated text is available in large quantities. Examples include large image collections with keywords, museum image collections with descriptive text, news photos, and images on the web. In this paper we discuss how the translation approach can give a handle on difficult questions such as: What counts as an object? Which objects are easy to recognize and which are hard? Which objects are indistinguishable using our features? How to integrate low level vision processes such as feature based segmentation, with high level processes such as grouping. We also summarize some of the models proposed for translating from visual information to text, and some of the methods used to evaluate their performance.
BibTeX
@inproceedings{barnard2003lt,
  author = {Kobus Barnard and Pınar Duygulu and David Forsyth},
  title = {{\<title\>Recognition as Translating Images into Text\</title\>}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2003},
  doi = {10.1117/12.478427},
}
Figure