Domain-Independent Captioning of Domain-Specific Images

Rebecca MasonNAACL 2013

TL;DR

This paper introduces a novel framework for generating image captions that doesn't rely on traditional visual recognition systems. Instead, it leverages domain-specific image-caption pairs to learn a topic model, which then helps rewrite existing captions to better describe new images within that domain. This approach aims to overcome the limitations of general-purpose captioning models that struggle with the vastness and specificity of real-world image data.

What Problem Does This Paper Address?

Automatically generating descriptive captions for images is a challenging task at the intersection of Computer Vision (CV) and Natural Language Processing (NLP). Existing methods often rely on supervised visual recognition systems trained on massive datasets to identify objects and scenes. However, this approach is limited by the relatively small number of visual classes that can be reliably recognized due to the immense effort required for hand-labeling. This leads to captions that are often generic and fail to capture the nuanced interpretations humans make. Furthermore, these systems assume that all relevant information is visually present, ignoring valuable contextual cues available in the surrounding text or metadata. The authors argue that this reliance on visual recognition imposes unrealistic constraints and hinders the generation of truly human-like captions.

Methods

The core of the proposed framework is a domain-specific approach that bypasses the need for explicit visual recognition of a predefined set of object classes. Instead, it trains a topic model on a corpus of images and their corresponding captions within a specific domain. This topic model jointly learns image features and natural language descriptions, capturing the latent thematic structure of the data.

The process involves:

  1. Sentence Extraction: For a given query image, the system first identifies visually similar images from the training set using a global image descriptor (GIST). Captions from these similar images are then extracted.
  2. Topic Modeling: A topic model, specifically a Polylingual Topic Model (similar to LDA), is trained on the image-caption pairs. This model treats images and text as two different "languages," learning to associate visual features with textual concepts. For a query image, it estimates a topic distribution.
  3. Sentence Compression: An Integer Linear Program (ILP) is used to compress the extracted caption. This ILP leverages the prior probabilities of words in captions and the posterior probabilities of words given the query image (estimated by the topic model). The objective is to maximize the accuracy of the compressed caption by deleting words that are less relevant to the query image, while also ensuring grammatical correctness through syntactic constraints.

Datasets

The framework is implemented and evaluated using the Attribute Discovery Dataset, specifically the women's shoes section. This dataset contains over 14,000 images of women's shoes with associated product descriptions. The authors note that this dataset offers a wide variety of visual attributes (texture, shape, material, color) and that the shoes are typically posed against solid color backgrounds, simplifying the visual aspect.

Key Insights and Ideas

  • Domain-Specific Advantage: By focusing on a specific domain (e.g., shoes), the problem of visual recognition becomes significantly easier, and the implicit context within that domain helps in generating more relevant captions.
  • Leveraging Collocated Text: The paper highlights that even without explicit visual recognition, readily available text associated with images (like product descriptions, file names, or website text) provides crucial contextual information that can be exploited.
  • Topic Models for Joint Learning: Using topic models to jointly learn from image features and text allows the system to capture the underlying semantic relationships, enabling it to infer word relevance for a given image.
  • Caption Rewriting over Generation: Instead of generating captions from scratch, the approach focuses on refining existing, human-written captions, which can be more robust and less prone to grammatical errors.
  • Bridging the Semantic Gap: The work implicitly addresses the "semantic gap" by using contextual information and topic modeling to infer meaning beyond just the visual elements, aiming for a more human-like interpretation.

Key Results

The paper reports a 30-35% improvement in finding words from a held-out image caption compared to previous methods and baselines when using their topic model for annotation. Automatic metrics like BLEU and ROUGE also show statistically significant improvements in the accuracy of extracted and compressed captions over simple extraction alone. Table 1 provides qualitative examples demonstrating the model's ability to remove irrelevant visual attributes (e.g., "snake-embossed leather") while retaining relevant ones (e.g., "shimmering," "slingback," "round open toe").

Limitations and Future Work

The authors acknowledge that their current framework is implemented on a specific domain (women's shoes) and that extending it to natural images is a key direction for future work. Natural images present greater challenges due to variations in lighting, viewpoints, and object occlusion. They propose clustering training data based on visual scene domains (e.g., mountains, beaches) rather than just object categories.

Additionally, they suggest improving the text generation component by exploring methods that jointly extract and compress sentences, rather than relying on a single extracted caption. They also plan to investigate the effectiveness of different feature descriptors (global vs. local) for sentence extraction across various domains and explore connections between visual and textual features.

Summary updated

BibTeX
@inproceedings{mason2013domain,
  author = {Rebecca Mason},
  title = {{Domain-Independent Captioning of Domain-Specific Images}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure