Abstract
Communication can be more effective when several media (such as text, speech, or graphics) are integrated and coordinated to present information. This changes the nature of media-specific generation (e.g., language generation), which must take into account the multimedia context in which it occurs. This paper presents work on coordinating and integrating speech, text, static and animated 3D graphics, and stored images as part of several systems developed at Columbia University. A particular focus is the generation of presentations that brief a user on information of interest.
Introduction: Beyond Generation in a Vacuum
In the late 1990s, most automated systems generated text or graphics in isolation. However, human communication is inherently multi-modal; we point at charts while speaking or use captions to explain complex diagrams. If an AI generates a sentence without "knowing" what the accompanying image looks like, the result is often redundant, confusing, or poorly timed.
The challenge lies in creating a "media coordinator" that understands how the presence of one medium (like a 3D graphic) should fundamentally change the content and structure of another (like a spoken explanation). This is critical in "briefing" scenarios—such as a doctor receiving a post-surgery update—where information must be conveyed with extreme conciseness and zero ambiguity.
Coordinated Systems at Columbia University
COMET: Technical Maintenance
The COMET system uses a content planner and a media coordinator to link text captions with 3D graphics for military radio maintenance. It generates "cross-references" by querying the graphics component to see if an object is actually visible or hidden inside another part.
As seen in Figure 1, the system explicitly mentions the "holding battery cover plate" and directs the user to the "right picture" where it is highlighted, ensuring the text and visual cues are perfectly aligned.
MAGIC: Healthcare Briefings
The MAGIC system focuses on the temporal synchronization of speech and animation for medical briefings. Instead of generating a full sentence and then trying to time it, MAGIC produces a partial ordering of references. This allows the system to "negotiate" timing between the speech synthesizer and the graphics engine.
Figure 3 illustrates how speech segments like "Ms. Jones is an eighty-year-old..." are synchronized with the display of patient data. This coordination ensures that a medical chart highlights exactly when the corresponding data is mentioned.
CDNS: News Summarization
CDNS is a multi-document summarization system that selects images from the web to accompany generated text. It uses statistical analysis of surrounding text to classify unlabeled images, ensuring the visual content matches the generated summary.
Key Insights: Efficiency and Negotiation
- The "Efficiency" of Multi-modality: Speech can be more colloquial and shorter if a precise textual label is visible on screen. For example, the system might say "pacemaker" while the screen displays "ventricular pacemaker," reducing cognitive load without losing precision.
- Negotiation over Generation: Coordination is about negotiation. The language generator must provide multiple phrasing options so the media coordinator can pick the one that best fits spatial constraints or animation timing.
- Context-Aware Referencing: The way a system refers to an object must change based on visual style. If a graphic uses a "cutaway" to show an internal battery, the language component must explicitly mention that the view is a cutaway to help the user orient themselves.
Conclusion and Future Directions
The systems developed at Columbia demonstrated functional breakthroughs in dynamic content generation and temporal precision. While limited by the computer vision capabilities of the era—requiring CDNS to rely on text-based clues rather than pixel analysis—the work laid the groundwork for modern multi-modal AI. Future research directions include the use of prosody (tone and rhythm) to improve coordination and the challenge of generating "update" summaries that focus only on new information.
Summary updated