TL;DR
This paper introduces a novel approach to automatically constructing complex, multi-modal user interfaces by leveraging Artificial Intelligence knowledge bases and rule-based systems. The core idea is to model both the application domain and the interface capabilities, then use rules to dynamically generate displays that integrate various presentation modes like graphics, natural language, and forms, significantly reducing manual interface development effort.
What Problem Does This Paper Address?
Building sophisticated user interfaces, especially those that dynamically adapt and integrate multiple presentation modes (like graphics, text, menus, and forms), is a notoriously time-consuming and difficult task. Current estimates suggest that interface code can account for up to 60% of an application's total code. Existing tools often fall short when it comes to creating interfaces that can intelligently choose between different media and methods, minimize redundancy, and ensure a consistent meaning representation across all modes. This paper tackles the challenge of automating the creation of these complex, integrated user interfaces, aiming to make them more adaptable, easier to modify, and quicker to develop.
Methods
The authors propose a "model-driven presentation design system" that relies on two key AI techniques:
- Knowledge Representation: They use two systems:
- NIKL: A network knowledge-base system (descended from KL-ONE) to model the entities of the application domain (e.g., ships, their statuses, missions) and the interface world (e.g., windows, maps, text strings). These models are organized into IS-A hierarchies to represent class relationships.
- KL-TWO: A hybrid knowledge representation system that interfaces with NIKL and a reasoner called PENNI. PENNI manages assertions about the application domain and uses NIKL's taxonomic reasoning to deduce new facts.
- Rule-Based Presentation Design:
- Application Model: Defines the objects and actions within the application's view of the world.
- Interface Model: Defines the objects and actions of the interface itself, unifying different presentation modes.
- Presentation Rules: These rules map objects from the application model to objects in the interface model. They range from application-specific rules (e.g., how to display a "mobile" ship) to general rules (e.g., choosing between a menu or a fill-in-the-blank form based on the number of choices).
- Presentation Designer: This component interprets data asserted into PENNI, uses the domain and interface models, selects appropriate rules, and generates a description of the desired display. This description is then interpreted by device drivers to create the actual output.
Datasets
Not discussed in the paper. The paper focuses on a demonstration application for a Naval database reporting system, implying the use of real-world naval data, but no specific datasets are detailed.
Key Insights and Ideas
- Unified Meaning Representation is Crucial: The authors emphasize that for integrated multi-modal interfaces to work, all presentation modes must derive from a common, unified meaning representation. This is a core principle behind their model-driven approach.
- Dynamic Display Generation: Fixed, multi-modal displays are insufficient for rapidly changing environments. Their system dynamically produces displays on demand, adapting to the specific information being presented.
- Leveraging AI for Interface Design: The paper demonstrates that AI knowledge bases and rule technology are powerful tools for automating complex interface construction, moving beyond manual coding.
- Generalization and Enhancement: The model-driven approach inherently supports generalization and makes interfaces easier to modify and enhance over time, addressing a key limitation of traditional interface development.
- Integration of Diverse Modes: The system successfully integrates natural language text, maps, tables, menus, and forms, showing that these can be treated as part of a cohesive whole rather than separate entities.
Key Results
The paper presents a demonstration version of the "Integrated Interfaces" system that successfully creates integrated multi-modal displays for a Naval database reporting application. The system can:
- Display information using a combination of natural language text, maps with icons and string tags, tables, and forms.
- Handle input through menus, forms, and pointing.
- Dynamically create displays similar to those manually prepared for the Navy.
- Demonstrate the ability to automatically generate displays based on rules, such as showing a "Disabled-Ship" as a red icon with its course as orientation and schedule as a text tag.
- Showcase the system's ability to classify collections of data (e.g., "Pacific Situation") and apply specialized presentation rules for them.
- Generate input forms dynamically, adapting to the number of choices available (e.g., using a fill-in-the-blank form for many choices vs. a menu for few).
Limitations and Future Work
Limitations:
- The authors acknowledge that their current system is a demonstration and that the complexity of real-world applications may require further refinement.
- While they have implemented rules for specific scenarios, they note that for many requests, global considerations are needed that go beyond simple low-level rules. Their approach of classifying data collections offers a partial solution.
Future Work:
The authors outline several promising directions to enhance the system:
- User Model: Tailoring presentations to individual user preferences, potentially by reducing screen clutter for experienced users.
- Dialogue Model: Allowing presentations to be more closely aligned with the user's specific request and purpose, rather than just displaying data.
- Screen Model: Explicitly modeling the screen as a structured entity containing icons and text with real-world denotations, enabling relationships between screen elements and their meanings.
- Extending Knowledge Representation: Further integration with other AI systems and potentially more sophisticated reasoning capabilities.
Summary updated