Bidirectional Use of Knowledge in the Multi-modal NL Access System XTRA

Jürgen Allgayer, Roman M. Jansen-Winkeln, Carola Reddig, Norbert ReithingerIJCAI 1989

Presented at a 1989 conference on Speech and Natural Language, the XTRA (eXpert TRAnslator) system represents a significant milestone in the evolution of natural language interfaces. Developed during an era when most natural language (NL) systems were rigid and task-specific, XTRA introduced a modular, multi-modal architecture designed to bridge the gap between human users and complex expert systems. By separating linguistic knowledge from domain-specific world knowledge and employing these sources bidirectionally, XTRA offered a more flexible and intuitive way to navigate data-heavy environments, such as tax preparation.

The Challenge of "Brittle" Interfaces

In the late 1980s, NL interfaces faced a primary hurdle: domain dependence. Linguistic rules were often hard-coded into the logic of the underlying expert system. If a developer wanted to port an interface from a tax-calculating system to a medical diagnosis tool, they essentially had to rebuild the system from scratch.

Furthermore, these systems were typically unidirectional—optimized for either parsing input or generating output, but rarely both—and limited to a single mode of communication (text). XTRA was engineered to address these three core issues: domain dependence, lack of bidirectionality, and the inability to process non-verbal cues like pointing gestures.

Modular Knowledge Representation with SB-ONE

To achieve domain portability, the researchers decoupled knowledge into two distinct bases:

  1. Functional-Semantic Structure (FSS): This contains domain-independent linguistic knowledge, defining how verbs, nouns, and adjectives relate to one another grammatically and semantically.
  2. Conceptual Knowledge Base (CKB): This contains domain-specific world knowledge—for example, defining what a "tax form," a "commute," or a "deduction" actually represents in the real world.

Both bases are encoded using SB-ONE, a specialized knowledge representation language derived from KL-ONE. SB-ONE utilizes a taxonomic hierarchy of concepts, roles, and restrictions to create a structured map of information.

A portion of the Conceptual Knowledge Base (CKB) illustrating the relationships between actions like 'TO GO' and entities like 'VEHICLE' and 'CITY'.

To bridge these two knowledge bases, XTRA employs a "matcher." This tool maps linguistic patterns from the FSS onto individualized concepts in the CKB. This allows the system to understand that a user's mention of "commuting" in a sentence refers to a specific "motion-by-means" concept within the database, complete with its own set of required attributes like source, destination, and cost.

Multi-modal Integration: The TACTILUS Module

One of XTRA's most innovative features is its treatment of pointing gestures. Rather than viewing a gesture as a separate, parallel input, XTRA treats it as a "deictic modifier" to a linguistic phrase.

A dedicated module called TACTILUS handles these gestures. When a user clicks on the screen, TACTILUS maps the coordinates to a "Form Hierarchy" (FH), which represents the visual structure of the interface (e.g., specific boxes on a tax form). This FH is linked back to the CKB, allowing the system to resolve ambiguous terms. For instance, if a user says "from here" and points to a specific field, the system identifies the city associated with that field and refines the linguistic concept of "here" into a concrete geographic location.

Case Study: The Tax Advisory Prototype

The system's capabilities are best demonstrated through its prototype application: a German tax advisory expert system. In this scenario, the user interacts with a digital version of an annual tax adjustment form.

The XTRA prototype interface, demonstrating a multi-modal interaction where a user points to a specific field on a tax form while providing natural language input.

Consider the following interaction sequence:

  1. User Input: The user types, "Ich fahre täglich von hier (^) mit einem Bus nach Völklingen" ("I commute daily from here (^) by bus to Völklingen"), while pointing at the city "Saarbrücken" on the form.
  2. Resolution: TACTILUS resolves the deictic reference "here" to Saarbrücken by combining the gesture's location with the form's structural hierarchy.
  3. Inference: By consulting the CKB, the system identifies that the "commute" concept requires a "cost" attribute, which is missing from the user's statement.
  4. Bidirectional Generation: Using the same knowledge bases used for analysis, the generation component (POPEL) formulates a targeted follow-up: "Was kostet die Fahrt?" ("How much does the trip cost?").

Limitations and Future Directions

While groundbreaking, the 1989 prototype had clear limitations. It relied on expensive, specialized AI hardware (Symbolics 3640 workstations) and faced challenges with complex logical operators like quantifiers ("all," "some") and negative information.

Additionally, while XTRA could understand pointing, it could not yet point back. The authors proposed a future gesture generator called ZORA to allow the system to use visual cues in its own responses. Despite these constraints, XTRA's modular approach to knowledge and its integration of multi-modal input laid the groundwork for the sophisticated natural language assistants we use today.

Summary updated

BibTeX
@inproceedings{allgayer1989bidirectional,
  author = {Jürgen Allgayer and Roman M. Jansen-Winkeln and Carola Reddig and Norbert Reithinger},
  title = {{Bidirectional Use of Knowledge in the Multi-modal NL Access System XTRA}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1989},
}
Figure