Panmodal Information Interaction

Chirag Shah, Ryen W. White

Abstract

The emergence of generative artificial intelligence (GenAI) is transforming information interaction. For decades, search engines have provided results in a standard format, but the ability to chat with AI-based agents is changing how people consume information at scale. These two modalities—traditional search and AI-powered chat—currently coexist in search engines, either loosely or tightly coupled. We believe this creates an opportunity to re-imagine the search experience as a "panmodal" experience. Unlike monomodal experiences, panmodal experiences make multiple modalities available (multimodal), support transitions between them (crossmodal), and seamlessly combine them to tailor task assistance (transmodal). Based on a survey of over 100 individuals, we present a vision for the future of information interaction using multiple modalities and the emergent capabilities of GenAI.

For decades, the "10 blue links" of search engines like Google and Bing were the gold standard for information retrieval. However, the sudden rise of GenAI agents like ChatGPT has introduced a second, powerful modality: natural language chat. Currently, these two worlds often exist as separate silos or are "loosely coupled," such as a chat tab appearing next to a search bar.

Researchers from the University of Washington and Microsoft Research identify a significant gap in our understanding of how users choose between these modes. The goal is to move toward a "seamless flow" where systems are designed to support panmodal interaction rather than forcing users to pick one modality over another.

A Taxonomy for Panmodal Interaction

To better understand this shifting landscape, the authors propose a new taxonomy for information interaction based on the level of integration between modalities:

  • Monomodal: Using only one modality (e.g., search only).
  • Multimodal: Multiple modalities are available and used for a task, but remain distinct.
  • Crossmodal: Directly supporting the switch from one modality to another (e.g., starting in search and moving to chat to refine the result).
  • Transmodal: An integrated experience where the task spans multiple modalities simultaneously (e.g., a search engine finding sources while a chat agent drafts a report based on them).
The Panmodal Support Taxonomy: Multimodal, Crossmodal, and Transmodal interaction models.

Empirical Insights: Search for Finding, Chat for Doing

The researchers conducted a qualitative survey of 130 US-based participants to gather personal accounts of using search and chat for professional and personal tasks. The results revealed a clear divide in modality preference based on the nature of the task.

Users overwhelmingly prefer traditional search for informational and planning tasks, such as "Planning a vacation" (61%) or "Looking for event info" (58%). Conversely, chat-based modalities dominate for "understanding" and "creation" tasks, such as "Writing a work-related report" (60%) and "Explaining photosynthesis" (55%).

Distribution of user preferences for Search and Chat across various information tasks.

Switching Behavior and Mental Models

The survey revealed a significant "crossmodal" flow that is largely a one-way street. Users often start with a broad search and switch to chat when they need information tailored or made interactive. In a specific analysis of 123 respondents, 15 users started in search and switched to chat to finish their task, while zero users started in chat and switched back to search for the same task.

Despite the rise of GenAI, search maintains a "trust" advantage. It remains the go-to for "Trustworthy and Verified Sources" and "Real-Time/Location-Based Info." Chat, while powerful for synthesis, is often criticized for its difficulty in following complex instructions or its limited creativity on sensitive topics.

Future Directions

As mental models for GenAI continue to form, the authors call for new "routing mechanisms" that can automatically suggest the best modality for a specific step in a task. While current commercial systems are mostly multimodal, the future lies in transmodal systems that can seamlessly blend these capabilities. Furthermore, the vision for panmodality extends beyond text to include audio, vision, touch, and eye-gaze tracking, creating a truly integrated future for information interaction.

Summary updated

Abstract

The emergence of generative artificial intelligence (GenAI) is transforming information interaction. For decades, search engines such as Google and Bing have been the primary means of locating relevant information for the general population. They have provided search results in the same standard format (the so-called "10 blue links"). The recent ability to chat via natural language with AI-based agents and have GenAI automatically synthesize answers in real-time (grounded in top-ranked results) is changing how people interact with and consume information at massive scale. These two information interaction modalities (traditional search and AI-powered chat) coexist in current search engines, either loosely coupled (e.g., as separate options/tabs) or tightly coupled (e.g., integrated as a chat answer embedded directly within a traditional search result page). We believe that the existence of these two different modalities, and potentially many others, is creating an opportunity to re-imagine the search experience, capitalize on the strengths of many modalities, and develop systems and strategies to support seamless flow between them. We refer to these as panmodal experiences. Unlike monomodal experiences, where only one modality is available and/or used for the task at hand, panmodal experiences make multiple modalities available to users (multimodal), directly support transitions between modalities (crossmodal), and seamlessly combine modalities to tailor task assistance (transmodal). While our focus is search and chat, with learnings from insights from a survey of over 100 individuals who have recently performed common tasks on these two modalities, we also present a more general vision for the future of information interaction using multiple modalities and the emergent capabilities of GenAI.
BibTeX
@article{shah2025panmodal,
  author = {Chirag Shah and Ryen W. White},
  title = {{Panmodal Information Interaction}},
  journal = {Communications of the ACM},
  year = {2025},
  url = {https://openreview.net/forum?id=1awtFnX48W},
}
Figure