A Conversational Model of Multimodal Interaction in Information Systems

Adelheit Stein, Ulrich ThielAAAI 1993

Sure, here's a blog-style summary of the paper "A Conversational Model of Multimodal Interaction in Information Systems":

TL;DR

This paper introduces a novel framework for modeling multimodal interactions in information systems, treating user actions like mouse clicks as "dialogue acts" within a conversational metaphor. This approach aims to create more intuitive and cooperative user interfaces by integrating natural language and graphical components, leading to reduced complexity and improved user experience.

What Problem Does This Paper Address?

Contemporary information systems often present complex data through multimodal interfaces (text, graphics, forms). While direct manipulation of objects reduces the distance between user intent and system action, navigating large information spaces can be challenging. Existing approaches often treat user interactions as simple method invocations, neglecting the conversational nature of information seeking. This paper addresses the need for a more sophisticated model that can handle the temporal structure and topical coherence of cooperative information-seeking dialogues, integrating various interaction styles (natural language, graphics) into a unified conversational framework.

Methods

The authors propose a comprehensive model based on an extended notion of "dialogue acts," which can be realized through linguistic or non-linguistic means. Key components of their approach include:

  • Dialogue Acts: User inputs (mouse clicks, menu selections) are interpreted as dialogue acts, expressing a discourse goal rather than just method invocation. These acts affect the internal representation of the ongoing dialogue.
  • Conversational Roles Network (COR): A recursive state-transition network models the temporal structure of dialogues. It defines dialogue states and transitions (dialogue contributions) between them, resembling a network of possible dialogue paths. Parameter A represents the information seeker, and B the information provider.
  • Hybrid Interaction Style: The model integrates graphical and natural language components, allowing the system to respond with visualizations of new graphical objects combined with natural language "comments."
  • Information-Seeking Strategies: A strategic level of description specifies the topical structure according to information-seeking strategies, enabling the system to plan dialogue content and suggest cooperative responses.
  • Case-Based Reasoning (CBR): The MERIT system utilizes a case-based dialogue manager that stores past interaction "cases" to guide users through current sessions by suggesting thematic progressions.

Datasets

Not discussed in the paper. The paper focuses on a theoretical model and its application in a prototypical system (MERIT).

Key Insights and Ideas

  • User Actions as Dialogue Acts: The core insight is re-framing direct manipulation actions (like mouse clicks) not as isolated commands but as communicative acts within a dialogue. This allows for a richer interpretation of user intent and more flexible system responses.
  • Conversational Metaphor for Multimodality: Applying the "conversation metaphor" to multimodal interfaces provides a unified framework for understanding and managing interactions that blend language and graphics.
  • Integrating Meta-Dialogic Options: The model effectively integrates complex dialogue control options (like "help," "history," "change case") as natural parts of the conversation, rather than as separate, potentially disruptive, interface elements.
  • Reducing Interface Complexity: By modeling dialogues conversationally, the system can preserve the richness of advanced direct manipulation interfaces while reducing their perceived complexity for the user.

Key Results

The paper describes the application of their model in the MERIT system, a prototypical knowledge-based user interface. The implementation led to:

  • Reduced User Interface Complexity: The system successfully reduced the complexity of the user interface.
  • Preservation of Dialogue Options: It preserved most of the useful, albeit sometimes confusing, dialogue options found in advanced direct manipulation interfaces.
  • Cooperative Interaction: The model facilitates cooperative system responses, enhancing the user's information-seeking experience.

Limitations and Future Work

  • Implicit Contributions: While the model allows for "implicit" dialogue contributions (where an act can be skipped if inferable from context), the paper notes that this requires careful handling to avoid ambiguity.
  • Thematic Level Specification: The COR network primarily addresses the illocutionary structure. The paper acknowledges the need for a prescriptive addition to specify the thematic level for more robust dialogue planning.
  • Future Work: The authors suggest that their approach can be extended to other domains and that further research into integrating different interaction styles within this conversational framework is warranted. The MERIT system's case-based approach to user guidance is also presented as a promising direction.

Summary updated

BibTeX
@inproceedings{stein1993conversational,
  author = {Adelheit Stein and Ulrich Thiel},
  title = {{A Conversational Model of Multimodal Interaction in Information Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1993},
}
Figure