Annotation Tools and Knowledge Representation for a Text-To-Scene System

Bob Coyne, Alex Klapheke, Masoud Rouhizadeh, Richard Sproat, Daniel BauerCOLING 2012

TL;DR

This paper introduces VigNet, a novel lexical resource that bridges high-level semantic frames (like FrameNet) with low-level graphical primitives for text-to-scene generation. By defining "vignettes" and object "affordances," they enable more nuanced scene creation that captures the specific ways actions are performed and objects interact.

What Problem Does This Paper Address?

Creating 3D scenes from natural language descriptions is challenging because language is often ambiguous and underspecified for visual representation. A simple verb like "wash" can imply vastly different visual scenarios depending on its arguments (e.g., washing an apple versus washing a floor). Existing systems often struggle to capture these subtle differences, leading to generic or inaccurate scene generation. This paper addresses the need for a richer knowledge base that maps linguistic actions and their arguments to specific visual representations, going beyond simple object placement to capture the dynamics of interactions and spatial relationships.

Methods

The core contribution is the development of VigNet, a lexically-oriented knowledge base. VigNet uses vignettes, which are specialized frames that represent the graphical realization of actions and compound objects. These vignettes act as an intermediary between high-level semantic frames (like those in FrameNet) and low-level graphical primitives.

Key methodological components include:

  • Primitive Graphical Frames: A set of recurring, low-level graphical relations (e.g., position, orientation, surface attributes) used to describe visual scenes.
  • Affordances: Properties of 3D objects that define how they can be interacted with (e.g., a doorknob's "OPEN-CLOSE-GRIP" affordance). These are crucial for applying graphical primitives to actual objects.
  • Vignette Definition: Vignettes are defined to map verb-argument tuples (extracted from text) to sets of primitive graphical frames. This allows for capturing variations in action execution based on the verb's arguments.
  • Annotation Tools: The authors developed web-based interfaces for annotating both location vignettes (using crowdsourcing via Amazon Mechanical Turk and their own WordsEye system) and action vignettes.

Datasets

  • British National Corpus (BNC): Used as a source for extracting verb-argument tuples for action vignette annotation.
  • WordsEye 3D Object Library: A library of approximately 2200 common 3D objects used for defining parts and affordances.
  • Gold-standard set of objects: Used to evaluate the precision and recall of object collection for location vignettes.

The paper also describes the creation of a new resource, VigNet, which is a lexically-oriented knowledge base populated with location and action vignettes.

Key Insights and Ideas

  • Vignettes as a Semantic-Graphical Bridge: The concept of vignettes is powerful because it explicitly links abstract semantic roles (from FrameNet) to concrete graphical representations, addressing the "form vs. function" gap in scene generation.
  • Affordances are Crucial for Interaction: Understanding an object's affordances is essential for accurately depicting how it can be used in an action. This moves beyond just knowing an object is a chair to knowing it has a "SEAT" affordance for sitting.
  • Contextual Nuance for Verbs: The paper highlights that verbs like "wash" require context from their arguments to be visually represented correctly. This necessitates a more granular knowledge representation than simple verb definitions.
  • Crowdsourcing and Self-Annotation: The authors demonstrate the effectiveness of both crowdsourcing (AMT) and using their own text-to-scene system (WordsEye) for annotating location vignettes, offering complementary benefits.
  • Focus on Graphical Constraints: Vignettes emphasize graphical constraints active at a single moment, prioritizing contextual entailment over complex causal or temporal reasoning, which simplifies the annotation and generation process.

Key Results

The paper details the development of the VigNet resource, which includes:

  • A defined set of primitive graphical frames and object affordances.
  • Approximately 50 detailed location vignettes created using WordsEye.
  • Annotation of about 90 core verbs with action vignettes, resulting in 450 top-level vignettes and 2500 inherited vignettes.
  • Successful annotation of their entire 3D object library, renaming about 18,000 parts and assigning 2,400 affordances.
  • Evaluation of location vignette annotation showed 91% precision and 88% recall for object collection.

Limitations and Future Work

  • Handling Unknown Words and Ambiguity: The current annotation process relies on annotators typing words, leading to potential issues with unknown words and word sense ambiguity. Future work involves a semi-automatic pass to normalize and disambiguate these.
  • Mapping to FrameNet: While vignettes are inspired by FrameNet, the direct mapping is bypassed for efficiency. Future work aims to link vignettes back to their corresponding FrameNet frames.
  • Generalization and Composition: A key challenge is generalizing vignettes to handle inputs not explicitly defined and composing action vignettes with location vignettes. The authors plan to leverage semantic distance estimation and ontology information for generalization, and to use default venues and affordance unification for composition.
  • Completeness of Vignettes: It's acknowledged that a vignette may not exist for every possible input. The system needs to generalize and find the "closest" vignette.

Summary updated

BibTeX
@inproceedings{coyne2012annotation,
  author = {Bob Coyne and Alex Klapheke and Masoud Rouhizadeh and Richard Sproat and Daniel Bauer},
  title = {{Annotation Tools and Knowledge Representation for a Text-To-Scene System}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure