Abstract
The emergence of generative artificial intelligence (GenAI) is transforming information interaction. For decades, search engines have provided results in a standard format, but the ability to chat with AI-based agents is changing how people consume information at scale. These two modalities—traditional search and AI-powered chat—currently coexist in search engines, either loosely or tightly coupled. We believe this creates an opportunity to re-imagine the search experience as a "panmodal" experience. Unlike monomodal experiences, panmodal experiences make multiple modalities available (multimodal), support transitions between them (crossmodal), and seamlessly combine them to tailor task assistance (transmodal). Based on a survey of over 100 individuals, we present a vision for the future of information interaction using multiple modalities and the emergent capabilities of GenAI.
Introduction: Beyond the "10 Blue Links"
For decades, the "10 blue links" of search engines like Google and Bing were the gold standard for information retrieval. However, the sudden rise of GenAI agents like ChatGPT has introduced a second, powerful modality: natural language chat. Currently, these two worlds often exist as separate silos or are "loosely coupled," such as a chat tab appearing next to a search bar.
Researchers from the University of Washington and Microsoft Research identify a significant gap in our understanding of how users choose between these modes. The goal is to move toward a "seamless flow" where systems are designed to support panmodal interaction rather than forcing users to pick one modality over another.
A Taxonomy for Panmodal Interaction
To better understand this shifting landscape, the authors propose a new taxonomy for information interaction based on the level of integration between modalities:
- Monomodal: Using only one modality (e.g., search only).
- Multimodal: Multiple modalities are available and used for a task, but remain distinct.
- Crossmodal: Directly supporting the switch from one modality to another (e.g., starting in search and moving to chat to refine the result).
- Transmodal: An integrated experience where the task spans multiple modalities simultaneously (e.g., a search engine finding sources while a chat agent drafts a report based on them).
Empirical Insights: Search for Finding, Chat for Doing
The researchers conducted a qualitative survey of 130 US-based participants to gather personal accounts of using search and chat for professional and personal tasks. The results revealed a clear divide in modality preference based on the nature of the task.
Users overwhelmingly prefer traditional search for informational and planning tasks, such as "Planning a vacation" (61%) or "Looking for event info" (58%). Conversely, chat-based modalities dominate for "understanding" and "creation" tasks, such as "Writing a work-related report" (60%) and "Explaining photosynthesis" (55%).
Switching Behavior and Mental Models
The survey revealed a significant "crossmodal" flow that is largely a one-way street. Users often start with a broad search and switch to chat when they need information tailored or made interactive. In a specific analysis of 123 respondents, 15 users started in search and switched to chat to finish their task, while zero users started in chat and switched back to search for the same task.
Despite the rise of GenAI, search maintains a "trust" advantage. It remains the go-to for "Trustworthy and Verified Sources" and "Real-Time/Location-Based Info." Chat, while powerful for synthesis, is often criticized for its difficulty in following complex instructions or its limited creativity on sensitive topics.
Future Directions
As mental models for GenAI continue to form, the authors call for new "routing mechanisms" that can automatically suggest the best modality for a specific step in a task. While current commercial systems are mostly multimodal, the future lies in transmodal systems that can seamlessly blend these capabilities. Furthermore, the vision for panmodality extends beyond text to include audio, vision, touch, and eye-gaze tracking, creating a truly integrated future for information interaction.
Summary updated