Generating Expressions that Refer to Visible Objects

Margaret Mitchell, Kees van Deemter, Ehud ReiterNAACL 2013

TL;DR

This paper introduces a novel algorithm for generating human-like referring expressions for visible objects, moving beyond simply identifying objects to mimicking how people naturally describe them. The algorithm accounts for over- and underspecification, and uses a stochastic approach to capture speaker variation, outperforming existing methods in evaluations.

What Problem Does This Paper Address?

Referring Expression Generation (REG) is the task of creating phrases that uniquely identify an object in a scene. Traditional REG algorithms focus on selecting properties that distinguish a target object from all other "distractor" objects. However, human language use, especially in visual descriptions, often deviates from this strict rule. People frequently use "overspecified" descriptions (including redundant properties) or "underspecified" ones (where the description might not uniquely identify the object). This paper addresses this gap by proposing an algorithm that mimics human-like descriptive tendencies, incorporating factors like visual salience and speaker variation, rather than solely focusing on logical uniqueness. The goal is to generate expressions that sound natural and are informed by how humans perceive and describe the visual world, bridging the gap between computer vision and natural language generation.

Methods

The core of the proposed "Visible Objects Algorithm" is a stochastic approach to selecting properties. Instead of deterministically picking properties to rule out distractors, it uses prior likelihoods of attributes (e.g., how likely is an object to be described by its color) and a penalty for longer descriptions to stochastically build a set of properties. This allows for generating multiple possible descriptions for the same object, reflecting human variation.

Key methodological aspects include:

  • Separation of Absolute and Relative Properties: The algorithm distinguishes between properties directly perceivable by computer vision (like color) and those requiring more reasoning (like size and location).
  • Stochastic Property Selection: Using prior likelihoods ($\alpha_{att}$) and a length penalty ($\gamma$), the algorithm probabilistically adds properties to a set. This is a departure from deterministic algorithms.
  • Scan-through Mechanism: After initial property selection, the algorithm simulates how humans might scan a scene, comparing the target object to others and adding properties if they help distinguish it, but without the strict "rule-out" objective.
  • Non-deterministic Evaluation: To handle the algorithm's stochastic nature, two evaluation methods are proposed:
  • Aligned Dice: Finds the best alignment between generated and observed property sets using a Dice similarity metric.
  • Majority: Compares the most frequent generated property set against the most frequent observed property set.

Datasets

The paper evaluates its algorithm on two existing REG corpora:

  • GRE3D3 Corpus: Used for evaluating type, color, size, and location.
  • TUNA Corpus: Used for evaluating type, color, size, and orientation.

The authors note that these corpora provide expressions elicited for computer-generated objects, making them suitable for evaluating reference to visible objects. They do not introduce new datasets.

Key Insights and Ideas

  • Human descriptions are not always minimal: The observation that people often over- or underspecify is crucial. This paper argues that a good REG system should mimic this, rather than striving for strict logical uniqueness.
  • Visual salience matters: Properties that "pop out" visually (like color, then size) are prioritized, mirroring human perception. This suggests a link between vision processing and language generation.
  • Stochasticity captures variation: The non-deterministic nature of the algorithm is key to capturing the fact that different people (or even the same person at different times) might describe the same object differently.
  • Evaluation needs to account for variation: Traditional deterministic metrics are insufficient for stochastic REG. The proposed "Aligned Dice" and "Majority" metrics offer ways to fairly compare non-deterministic systems with human data.

Key Results

The Visible Objects Algorithm generally outperforms the Incremental Algorithm (IA) and the Graph-Based Algorithm (GB) on both corpora and evaluation metrics.

  • GRE3D3: Achieved higher accuracy with ALIGNED DICE (around 88-90%) compared to IA (87-88%) and GB (75-87%). In the MAJORITY metric, it was competitive, predicting the majority property set in 40% of test scenes.
  • TUNA: Showed similar competitiveness, with ALIGNED DICE scores around 86-88% for the proposed algorithm, compared to IA (81-87%) and GB (66-75%). GB performed poorly in the TUNA corpus, possibly due to data sparsity issues.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Limited Property Set: The current evaluation focuses on a subset of properties (color, size, location, orientation). Future work should incorporate more complex absolute properties like shape and material, as well as object parts.
  • Interconnected Properties: The algorithm doesn't currently model how properties might be interconnected (e.g., a wooden object is likely to be described as "wooden" rather than by its color).
  • Typicality and Atypicality: The role of typicality (selecting properties because they are typical or atypical) is not yet modeled.
  • More Sophisticated Models: Future work could involve building more sophisticated graphical models and exploring conditional probabilities for generating properties based on previous context.
  • Better Parameter Estimation: Developing better methods for determining attribute likelihoods and length penalties is also an area for future research.
  • Naturalistic Objects: The current evaluation uses computer-generated objects. Extending the framework to handle naturalistic objects with more complex visual features is a significant future challenge.

Summary updated

BibTeX
@inproceedings{mitchell2013generating,
  author = {Margaret Mitchell and Kees van Deemter and Ehud Reiter},
  title = {{Generating Expressions that Refer to Visible Objects}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure