Tailoring Lexical Choice to the User's Vocabulary in Multimedia Explanation Generation

Kathleen R. McKeown, Jacques Robin, Michael A. TanenblattACL 1993 · OpenReview

Abstract: In this paper, we discuss the different strategies used in COMET (COordinated Multimedia Explanation Testbed) for selecting words with which the user is familiar. When pictures cannot be used to disambiguate a word or phrase, COMET has four strategies for avoiding unknown words. We give examples for each of these strategies and show how they are implemented in COMET.

Introduction

Developed in the early 1990s at Columbia University, COMET (COordinated Multimedia Explanation Testbed) represents a foundational advancement in Natural Language Generation (NLG). The system was designed to address a critical challenge in technical communication: tailoring instructions to a user’s specific vocabulary level while maintaining tight coordination between text and graphics. Unlike static documentation, COMET dynamically adjusts its language based on a user model, ensuring that technical instructions remain accessible to both experts and novices.

The COMET Architecture

COMET employs a sophisticated architecture that interleaves content planning with lexical choice. At its core is the Functional Unification Formalism (FUF), which allows the system to resolve semantic, syntactic, and pragmatic constraints simultaneously. This approach ensures that word choice is not treated as an isolated "find-and-replace" task but as a global decision that influences the entire structure of a sentence.

The COMET system architecture.

As shown in the architecture, the Lexical Chooser works in tandem with the Graphics Generator under the supervision of a Media Coordinator. This coordination allows the system to determine whether a concept is best explained through text, an illustration, or a combination of both.

Strategies for Handling Unknown Words

In technical domains like military radio maintenance, experts and novices use vastly different vocabularies. COMET identifies that changing a single noun often requires a "ripple effect" of rephrasing verbs and prepositions to maintain grammatical correctness. To handle terms unknown to a user, COMET employs four distinct strategies:

  1. Simple Substitution: Replacing a technical term with a simpler synonym (e.g., "arbitrary number" becomes "some number").
  2. Conceptual Rephrasing: Decomposing a complex concept into physical actions. For instance, instead of "Check the polarity," a novice might be told: "Make sure the plus on the battery lines up with the plus on the battery compartment."
  3. Descriptive Expressions: Using spatial relations or physical attributes, such as referring to a "COMSEC cable" as "the cable that runs to the RT."
  4. Discourse History: Leveraging the context of previous actions, such as referring to an object as "the cable you just removed."

Multimedia Coordination and "Pictures as Definitions"

A key insight of the COMET project is that graphics can serve as visual definitions. If the system identifies that a user is unfamiliar with a term like "FCTN knob," it can generate an illustration to disambiguate the object, allowing the text to remain concise.

A multimedia explanation for setting a knob.

If a suitable picture does not already exist in the discourse history, the Lexical Chooser can "call back" to the content planner to request a new illustration. This "wait-and-see" strategy prevents the system from making premature lexical choices that might conflict with the available visual aids or the necessary sentence syntax.

Conclusion and Impact

Operating on a knowledge base of military radio repair (KY57 and RT units), COMET demonstrated that a unification-based approach could successfully manage the complexities of real-world technical rephrasing. While the manual labor required to scale such a system was significant, COMET's core principles—treating lexical choice as a global constraint and integrating graphics as a first-class citizen in the lexicon—remain highly relevant to modern intelligent tutoring and automated documentation systems.

Summary updated

Abstract

In this paper, we discuss the different strategies used in COMET (COordinated Multimedia Explanation Testbed) for selecting words with which the user is familiar. When pictures cannot be used to disambiguate a word or phrase, COMET has four strategies for avoiding unknown words. We give examples for each of these strategies and show how they are implemented in COMET.
BibTeX
@inproceedings{mckeown1993tailoring,
  author = {Kathleen R. McKeown and Jacques Robin and Michael A. Tanenblatt},
  title = {{Tailoring Lexical Choice to the User's Vocabulary in Multimedia Explanation Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1993},
  url = {https://openreview.net/forum?id=HJEX-3ldWr},
}
Figure