Understanding Descriptions of Visual Scenes Using Graph Grammars

Daniel BauerAAAI 2013 · DOI

Abstract

Generating 3D scenes from natural language requires a deep understanding of the spatial and physical implications of abstract verbs. This paper proposes a framework for text-to-scene generation that utilizes Synchronous Hyperedge Replacement Grammars (SHRG) to bridge the gap between linguistic predicate-argument structures and concrete 3D environments. By decomposing high-level functional descriptions into low-level spatial primitives, the system creates a formal, multi-layered representation that can be rendered into realistic 3D models.

The Challenge of Spatial Grounding

While modern Natural Language Processing (NLP) systems excel at identifying "who did what to whom," they often lack the depth required for visual tasks. For instance, the sentence "A man is sitting at a desk" identifies the entities and the action, but does not inherently specify the spatial layout: the position of the hands, the distance to the computer, or the physical support of the chair. Existing systems like WordsEye often rely on manual, brittle rules to bridge this gap. This work addresses the need for a formal, automated method to translate functional linguistic meaning into the detailed spatial constraints necessary for 3D modeling.

Methodology: Multi-Layered Graph Grammars

The proposed pipeline leverages Hyperedge Replacement Grammars (HRG) and Frame Semantics to represent and expand scene descriptions:

  1. Multi-Level Meaning Representation: The system generates graphs at varying levels of granularity. High-level graphs capture functional aspects, while low-level graphs detail conceptual primitives and spatial relations (e.g., "on," "near," "supported by").
  2. Synchronous HRGs (SHRG): A grammar is employed that pairs linguistic string rules with corresponding graph rules. Parsing a sentence simultaneously constructs a graph fragment representing its meaning.
  3. Graph Rewriting: HRG rules are used to "rewrite" or expand high-level nodes into detailed low-level fragments. For example, a node representing "sitting" is expanded into a sub-graph of spatial constraints between the agent and the furniture.
The expansion of a high-level functional graph (a) into a low-level spatial graph (b) using graph grammars.
  1. VigNet Integration: The system utilizes VigNet, a knowledge base extending FrameNet, to provide the "recipes" for decomposing complex frames into visual primitives.
  2. Constraint Satisfaction: To ensure physical validity, the system employs constraint solvers to resolve potential conflicts, such as object interpenetration, finding the most valid spatial configuration.

Data and Resources

The research utilizes several key datasets and tools:

  • WordsEye Corpus: A collection of 13,000 low-level textual descriptions paired with rule-generated scene representations.
  • VigNet: A specialized lexical resource for frame decomposition.
  • New Annotation Efforts: The author is developing tools to create a new dataset of high-level scene descriptions paired with intended 3D layouts, utilizing crowdsourcing to ensure natural language diversity.

Insights and Preliminary Results

A core insight of this work is that scene understanding is a process of expansion. A single verb acts as a compressed representation that must be "unzipped" into numerous spatial constraints. Directed graphs are uniquely suited for this task, as they can represent the hierarchical nature of language and coreference more effectively than traditional tree structures.

A 3D scene rendered from the spatial constraints generated by the graph grammar pipeline.

Preliminary results focus on the feasibility of the SHRG approach. The author has successfully demonstrated the automatic acquisition of SHRG rules from data and experimented with toy grammars capable of parsing limited scene descriptions into multi-layered graphs.

Limitations and Future Directions

Current limitations include scalability to complex domestic scenes and the resolution of spatial ambiguities. Future work will focus on expanding the grammar to handle more diverse environments and utilizing the WordsEye rendering engine for human-based end-to-end evaluation of generated scenes.

Summary updated

BibTeX
@inproceedings{bauer2013understanding,
  author = {Daniel Bauer},
  title = {{Understanding Descriptions of Visual Scenes Using Graph Grammars}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2013},
  doi = {10.1609/aaai.v27i1.8498},
}
Figure