Developing the COMMENTATOR, A Computer System Simulating Verbal Production

Milan Bily, Bengt SigurdCOLING 1982

TL;DR

This paper introduces "COMMENTATOR," an early system for generating natural language descriptions of dynamic visual scenes. It tackles the challenge of producing contextually relevant and non-redundant commentary by simulating human-like language production, moving beyond simple template-based generation.

What Problem Does This Paper Address?

The paper addresses the fundamental challenge of creating computer systems that can generate natural language descriptions of dynamic visual environments. Traditional approaches often relied on pre-written sentences or simple templates, which lack the flexibility and contextual awareness of human commentary. The authors aim to develop a system that can produce "human-like texts" by simulating the cognitive processes involved in language production, specifically focusing on how humans observe and describe events. This involves not just identifying objects and their positions but also understanding relationships, inferring actions, and avoiding redundant or obvious statements.

Methods

The core of the "COMMENTATOR" system involves a multi-stage process:

  • Scene Representation: The system starts with primary information about the scene, such as the coordinates of objects (two persons, Adam and Eve, and a gate).
  • Deriving Complex Values: From these raw coordinates, the system derives more complex relational information like distances and relative positions ("to left," "to right").
  • Abstract Predicate Generation: Based on a "question menu," the system tests conditions for using abstract predicates (e.g., "is to the left of"). This results in a set of positive or negative abstract sentences.
  • Syntactic Structuring: These abstract sentences are then structured into subject-predicate-object forms. Connective elements ("also," "either," "yet") are added to link propositions.
  • Surface Phrase Generation: Proper names and pronouns are chosen based on reference relations to the preceding text. Finally, abstract propositions are converted into surface phrases and words.

A more advanced system, "COMMIE," is also discussed, which aims to overcome "COMMENTATOR's" shortcomings by incorporating a larger vocabulary, more dynamic scene simulation, and a focus on avoiding redundancies and implied information. COMMIE uses a two-dimensional array to store "mini-thoughts" (predications) and simulates forgetting by gradually letting these contents pass a "horizon of forgetting." It also aims to incorporate theories of Functional Sentence Perspective (FSP) for more nuanced commentary.

Datasets

Not discussed in the paper. The system generates its own dynamic scenes on a micro-computer monitor.

Key Insights and Ideas

  • Simulating Language Production: The paper emphasizes that effective commentary requires simulating the cognitive processes of language production, not just pattern matching.
  • Contextual Relevance: Commentary should be meaningful as an answer to expectations evoked by the previous context, rather than stating obvious facts.
  • Avoiding Redundancy: A key challenge is to avoid stating information that is already implied by previous utterances (e.g., if "Adam is to the left of Eve" is stated, "Eve is to the right of Adam" is redundant).
  • Dynamic Scene Understanding: The system moves beyond static descriptions to interpret and comment on evolving scenes, including movement and approaching goals.
  • Abstract Predicates: The use of abstract predicates as an intermediate step allows for more flexible and generalizable descriptions.

Key Results

The paper provides an example of the text produced by "COMMENTATOR," demonstrating its ability to generate a sequence of related sentences describing the movement of Adam and Eve relative to each other and a gate. The example text includes descriptions of relative positions, approaching actions, and proximity. The authors state that "COMMIE" aims for a vocabulary ten times larger and more sophisticated conditions for generating commentary.

Limitations and Future Work

  • Complexity of Real-World Scenes: The authors acknowledge that commenting on arbitrary, complex scenes with many objects and actions would require sophisticated pattern recognition, which is a difficult problem.
  • Future Versions: Future work includes developing a version that produces spoken comments using speech synthesizers, aiming for better prosody than standard text-to-speech systems. They also plan to explore contrastive accents derived from the system's knowledge of preceding statements and syntactic structure.
  • Practical Applications: While practical applications like automatic radar commentary are suggested, the authors note that these have been little considered so far, though they envisage computer-generated verbal comments becoming common in future robotic systems.

Summary updated

BibTeX
@inproceedings{bily1982developing,
  author = {Milan Bily and Bengt Sigurd},
  title = {{Developing the COMMENTATOR, A Computer System Simulating Verbal Production}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1982},
}
Figure