This 1991 classic from the AAAI proceedings represents a foundational moment in multimodal AI. Long before modern Large Language Models (LLMs) were "multimodal" by training on massive image-text datasets, researchers were exploring how to use symbolic logic and hierarchical planning to enable computers to coordinate speech and graphics in real-time.
The Challenge of Multimodal Coordination
In the early 1990s, AI systems were beginning to excel at either Natural Language Generation (NLG) or graphical rendering, but rarely both in a synchronized fashion. A typical system might provide a block of text alongside a static map, but the two components lacked mutual awareness.
The central challenge addressed by Mark Maybury is coordination. For a system to provide effective route directions, it must know when to zoom in on a map, when to highlight a specific road, and how to generate sentences that refer precisely to what is currently visible on the display. The goal was to move beyond "canned" explanations toward a system that reasons about the purpose of communication and selects the optimal mix of media to achieve that goal.
A Framework of Communicative Acts
The core of this approach is a Hierarchical Planner that treats communication as a goal-directed activity. Maybury extends traditional speech act theory to include visual and rhetorical actions, breaking communication down into three distinct levels:
- Rhetorical Acts: Media-independent goals such as Identify, Describe, Compare, or Narrate.
- Linguistic Acts: The building blocks of speech, including Illocutionary acts (intent, like "Request") and Locutionary acts (surface form, like "Assert").
- Visual Acts: Actions performed on a display, including Deictic acts (pointing/highlighting), Display Control (zooming/panning), and Depict acts (drawing lines or icons).
Goal-Directed Planning with Operators
The system utilizes Plan Operators defined by specific constraints, preconditions, and effects. When the system identifies a communicative goal—for example, identifying a location—the planner evaluates the current state of the display and the knowledge of the user.
If a goal is to Identify a location, the planner checks if the entity is currently visible. If it is not, the planner generates a sub-goal to perform a "Zoom" or "Pan" action. Once the preconditions are met, the act is decomposed into synchronized visual and linguistic components. For instance, the system might simultaneously highlight a town on the map and generate the text: "Karl-Marx-Stadt is a town located at..."
Implementation and Key Insights
The framework was implemented within the Map Display System (MDS), which contained a rich knowledge base of over 600 German towns, NATO airbases, and an extensive road network. Several key insights emerged from this work:
- Communication as Action: Showing a map and speaking a sentence are functionally equivalent; both are acts intended to modify the listener's mental state.
- Media Independence: High-level rhetorical goals allow the system to choose the most effective medium. The system uses "preference metrics" to favor multimodal output over text-only explanations when appropriate.
- Spatial Focus: The system tracks what is currently on screen to handle linguistic nuances, such as choosing between "this" and "that" based on visual proximity.
- The "Make-Visible" Sub-goal: The planner logically determines that it cannot point to an object that is off-screen, automatically triggering map adjustments before beginning a description.
Conclusion and Legacy
The paper successfully demonstrated a system capable of generating cohesive, multi-step route instructions. By tying linguistic realization to the visual state, the system ensured that directions like "Northeast" or "four kilometers" were calculated accurately from the underlying map data at the exact moment of generation. While limited by the primitive nature of 1990s graphical gestures, this work laid the groundwork for modern interactive assistants that seamlessly integrate visual and verbal information.
Summary updated