Multi-Modal References in Human-Computer Dialogue

Jeannette G. Neal, Zuzana Krifka Dobes, Keith E. Bettinger, Jong S. ByounAAAI 1988

Introduction

In the late 1980s, researchers at the Calspan-UB Research Center developed CUBRICON, a pioneering intelligent interface system designed to move beyond the limitations of uni-modal command-line interactions. Presented at AAAI-88, CUBRICON integrated natural language (NL) with deictic pointing gestures, creating a more fluid human-computer dialogue. By leveraging deep domain knowledge and a sophisticated discourse model, the system could resolve ambiguous or even erroneous user inputs in real-time.

The Challenge of Uni-modal Interfaces

Human communication is inherently multi-modal, relying on a synergy of speech, gestures, and visual aids. However, early computer systems were largely restricted to text. In high-stress environments like military tactical air control, describing complex objects solely through words (e.g., providing precise coordinates for a specific SAM site) is cognitively exhausting.

The CUBRICON interface displaying tactical information over a map of East and West Germany.

As shown in Figure 1, the CUBRICON display could become highly cluttered with various icons representing airbases, missile sites, and fuel storage facilities. The challenge was building a system that could synchronize words with mouse clicks, disambiguate overlapping icons, and correct user errors when verbal descriptions did not perfectly match the physical gesture.

System Architecture and Methods

Built on a Symbolics Lisp Machine, CUBRICON utilized several advanced AI techniques of its era:

  • Knowledge Representation: The system used the SNePS (Semantic Network Processing System) to store domain knowledge in an "AKO" (A Kind Of) hierarchy. This allowed the system to understand relationships, such as an "SA-2" being a specific type of "SAM system."
  • The Compound Stream: Input from speech recognizers, keyboards, and mice were fused into a single stream, preserving the temporal relationship between spoken words and physical gestures.
  • Discourse Modeling: CUBRICON maintained two types of focus: a Main Focus List for entities mentioned in conversation and a Virtual Display list of all objects currently visible on the screen.
  • ATN Parser: An Augmented Transition Network (ATN) parsed the compound stream to interpret noun phrases and locative adverbials in context.

Intelligent Referent Resolution

One of CUBRICON’s most innovative features was its ability to handle "fuzzy" or ill-formed input. If a user pointed at an empty space while saying "this SAM site," the system would search the vicinity of the click for the nearest object matching that description.

Furthermore, the system utilized contextual filtering. By analyzing the verb in a sentence (e.g., "based here"), CUBRICON used "case frames" to determine that the object of the gesture must be a location or institution, helping to filter out irrelevant icons in a crowded area.

Results and Legacy

The CUBRICON project demonstrated that multi-modal interaction could significantly reduce cognitive workload. It successfully handled three primary interaction types:

  1. Substitution: Replacing a noun phrase with a point ("What is the status of <point>?").
  2. Supplementation: Using a point to clarify a category ("this <point> airbase").
  3. Ambiguity Resolution: Correcting for near-misses or overlapping icons.

While limited by the hardware of the time—such as discrete speech recognition requiring pauses between words—CUBRICON remains a landmark in AI history. It proved that for computers to be truly intelligent partners, they must understand not just what we say, but the visual context of our interactions.

Summary updated

BibTeX
@inproceedings{neal1988multi,
  author = {Jeannette G. Neal and Zuzana Krifka Dobes and Keith E. Bettinger and Jong S. Byoun},
  title = {{Multi-Modal References in Human-Computer Dialogue}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1988},
}
Figure