TL;DR
This paper introduces Multimodal Functional Unification Grammar (MUG), a novel formalism for dynamically generating adaptive multimodal user interfaces. MUG allows systems to create coordinated outputs across different modalities (like voice and screen) that are tailored to the user's situation and preferences, moving beyond static or less flexible interface generation.
What Problem Does This Paper Address?
Traditional human-computer interaction often relies on separate input and output modalities, failing to capture the rich, synchronized communication seen in human-human interactions. While some systems attempt multimodal input or static multimodal documents, generating dynamic, coordinated multimodal output that adapts to the user and context remains a challenge. This paper addresses the need for user interfaces that can fluidly adapt their output across modalities (e.g., voice and screen) to provide complementary or redundant information, thereby increasing bandwidth and usability, especially in diverse situations and on different devices.
Methods
The core contribution is the Multimodal Functional Unification Grammar (MUG). MUG is a unification-based formalism, derived from Functional Unification Grammar (Kay, 1979), that generates coordinated multimodal content. Key aspects include:
- Functional Descriptions (FDs): MUG uses FDs to specify realization variants for partial semantic or syntactic representations. These FDs can be general or mode-specific.
- Unification: The grammar components are unified to build output. This process augments FDs with additional information, allowing for structure sharing and recursive definitions.
- Non-deterministic Generation: MUG generates multiple output variants for a given input. This ambiguity is crucial for adaptability.
- Adaptation and Evaluation: A comparative simple user and situation model, along with a heuristic function, evaluates these variants. The goal is to select the variant that maximizes utility to the user and minimizes cognitive load, considering factors like device capabilities and user context (e.g., driving a car).
- Coherence: The system aims for lexical consistency across modes (e.g., using "mobile phone" consistently) and complementary output where appropriate (e.g., a deictic expression in voice setting a feature for the screen).
Datasets
Not discussed in the paper.
Key Insights and Ideas
- Dynamic Generation is Key: The paper argues that interfaces should be generated "on the fly" to truly adapt to user needs and contexts, rather than relying on pre-defined structures.
- Unification Simplifies Integration: By using unification, MUG avoids the need for explicit, complex rules to integrate multimodal information, which was a limitation in prior work (e.g., Johnston, 1998).
- Adaptivity as a Trade-off: The system explicitly models adaptivity as a trade-off between realizing requested content, maximizing user utility, and minimizing cognitive load. This is formalized in an objective function.
- Mode-Specific Information Encapsulation: FDs allow for clear distinctions between information shared across all modes, mode-specific information, and information requiring inter-modal collaboration.
Key Results
The paper demonstrates MUG's functionality in the context of a personal assistant for sending email messages on a PDA-sized device. Examples show how MUG can generate different outputs based on the device and situation:
- A PDA with a touch-screen can utilize GUI widgets and voice.
- A mobile phone interface, when adapted, might de-emphasize screen output and increase the cost associated with it, reflecting a smaller screen and potentially a user in a more constrained situation.
- The system can generate redundant content for novice users and complementary content for experienced users.
Quantitative results are not detailed, but the paper emphasizes the qualitative benefits of adaptivity and coherence.
Limitations and Future Work
- Evaluation of Adaptivity: The authors acknowledge that further evaluation is needed to determine if the fitness function accurately reflects user satisfaction and if the adaptivity is truly advantageous for users on the go.
- Controlled User Trials: Controlled user trials are planned to evaluate usability, task completion times, user frustration, and satisfaction, as there is no established gold standard for dynamic multimodal interface generation.
- Broader Applications: The authors suggest future work could extend MUG to more complex systems, such as interfaces for navigating information-rich social networks or multilingual kiosks for public transportation.
- User Modeling: While a simple user and situation model is used, more sophisticated models could further enhance adaptivity.
Summary updated