TL;DR
This paper proposes that generating multimodal discourse (combining text, graphics, pointing, and animations) can be viewed as an incremental planning process, extending computational linguistics concepts to this richer communication domain. The authors introduce the WIP system, which generates context-sensitive multimodal explanations by interleaving content and mode selection, aiming for a multiplicative improvement in communication capabilities.
What Problem Does This Paper Address?
The paper addresses the challenge of effectively integrating different communication modalities (text, graphics, pointing, animations) to create richer and more effective multimodal discourse. Traditional approaches often treat these modalities separately, leading to a simple merging of outputs rather than a truly coordinated and synergistic communication. The authors argue that a more sophisticated approach is needed, one that considers how these modalities can mutually inform and enhance each other to achieve a communicative goal. This involves moving beyond simply verbalizing or visualizing information independently and instead planning how to best convey information across all available channels.
Methods
The core of the proposed solution is the WIP (Knowledge-Based Presentation System). WIP treats multimodal discourse generation as an incremental planning process. A key methodological innovation is the interleaving of content and mode selection. This contrasts with sequential architectures where content is planned first, then modes are selected, which the authors find problematic because content decisions are heavily influenced by how information will be presented.
WIP utilizes a parallel top-down planner and a constraint-based layout manager. The planning process starts with a high-level communicative act and breaks it down into elementary acts that can be realized by specific modalities. The system also incorporates seven discourse parameters (e.g., user stereotypes, target language, layout format, output modes) to tailor presentations to specific contexts, allowing for a large number of alternate multimodal presentations from the same content.
Datasets
Not discussed in the paper.
Key Insights and Ideas
- Multimodal Discourse as Incremental Planning: The central idea is that planning for multimodal communication is fundamentally similar to planning for text generation, but on a richer, more complex scale. This allows for a significant methodological transfer from computational linguistics.
- Methodological Transfer from NLP: Many concepts from natural language processing, such as speech acts, coherence, focus, and anaphora, gain an extended meaning and become crucial in the multimodal context.
- Multiplicative Improvement: The goal is not just to combine outputs from separate generators but to achieve a "multiplicative improvement" in communication by having modalities refer to and depend on each other.
- Interleaving Content and Mode Selection: This is a crucial departure from sequential models. The authors argue that the choice of modality strongly influences content decisions, and vice-versa, necessitating a tightly coupled planning process.
- Context-Sensitive Generation: The WIP system emphasizes generating presentations "on the fly" and customizing them for the specific discourse situation using contextual parameters, rather than relying on pre-authored hypermedia content.
- Layout as a Rhetorical Force: The paper acknowledges that layout itself plays a role in influencing the audience's attention and understanding.
Key Results
The paper describes the WIP system's ability to generate simple multimodal explanations in German and English for tasks like operating an espresso machine or assembling a lawn-mower. The system demonstrates language and application independence. The combinatorics of WIP's contextual parameters can generate 576 alternate multimodal presentations of the same content, highlighting its adaptability. The authors also mention follow-up work on the PPP (Personalized Plan-Based Presenter) project, which further addresses planning for presentation acts like pointing and coordinated speech.
Limitations and Future Work
The authors acknowledge that a sequential architecture for content planning and realization is inappropriate. They also note that it's impossible to anticipate all the needs of every user in every discourse situation, which is why their system prioritizes runtime adaptation. Future work, as indicated by the PPP project, focuses on planning specific multimodal presentation acts like pointing and coordinating speech output with visual displays. The paper suggests that the insights gained from systems like HAM-ANS, VITRA, XTRA, and WIP provide a strong foundation for understanding the complex interdependencies in coordinated multimodal discourse.
Summary updated