Panmodal Information Interaction

Ryen W. White, Chirag Shah

This paper, recently published by researchers from the University of Washington and Microsoft Research, explores the shifting landscape of how we find and use information in the age of Generative AI (GenAI).

TL;DR

The paper introduces the concept of panmodal information interaction, moving beyond simple "search vs. chat" to a future where multiple modes of interaction are seamlessly integrated. Based on a user survey, the authors argue that while search is for "finding," chat is for "doing," and the next generation of AI systems must support the fluid transitions (crossmodal) and deep integration (transmodal) between these modes.

What Problem Does This Paper Address?

For decades, the "10 blue links" of search engines like Google and Bing were the gold standard for information retrieval. The sudden rise of GenAI agents like ChatGPT has introduced a second, powerful modality: natural language chat.

Currently, these two worlds often exist as separate silos or are "loosely coupled" (e.g., a chat tab next to a search bar). The authors identify a gap in our understanding of how users choose between these modes, why they switch from one to the other, and how systems can be designed to support a "seamless flow" rather than forcing users to pick one or the other.

Methods

The authors conducted a survey of several hundred crowdworkers (resulting in 130 high-quality, analyzed responses) to gather personal accounts of using search and chat for professional and personal tasks.

They used these insights to propose a new taxonomy for information interaction:

  • Monomodal: Using only one modality (e.g., search only).
  • Multimodal: Multiple modalities are available and used for a task.
  • Crossmodal: Directly supporting the switch from one modality to another (e.g., starting in search and moving to chat to refine the result).
  • Transmodal: An integrated experience where the task spans multiple modalities simultaneously (e.g., a search engine finding sources while a chat agent drafts a report based on them).

Datasets

The primary data source is a qualitative survey of 130 US-based, English-speaking participants. The survey included:

  • Multiple-choice questions on modality preference for specific tasks.
  • Open-ended text responses describing successes and failures in search and chat.
  • Demographic data (60% male, 39% female, 1% other).

Key Insights and Ideas

  • The "Search for Finding, Chat for Doing" Divide: Users overwhelmingly prefer traditional search for informational and planning tasks (like finding a vacation spot or event info). Conversely, chat-based modalities dominate for "understanding" and "creation" tasks (like explaining scientific concepts or writing reports).
  • The One-Way Street: The survey revealed a significant "crossmodal" flow from search to chat. Users often start with a broad search and switch to chat when they need the information tailored or made interactive. Interestingly, in the survey data, almost no users switched from chat back to search for the same task.
  • Mental Models are Still Forming: While users have decades of experience with search, they are still experimenting with "prompt engineering" and forming mental models of what GenAI can and cannot do.
  • Search's "Trust" Advantage: Search is still the go-to for "Trustworthy and Verified Sources" and "Real-Time/Location-Based Info," whereas chat is criticized for its "Limited Creativity on Sensitive Topics" and "Difficulty Following Complex Instructions."

Key Results

  • Task Preferences:
    • Search: Preferred for "Planning a vacation" (61%) and "Looking for event info" (58%).
  • Chat: Preferred for "Writing a work-related report" (60%) and "Explaining photosynthesis" (55%).
  • Switching Behavior: In a specific analysis of 123 respondents, 15 users started in search and switched to chat to finish their task. Zero users started in chat and switched to search.
  • Qualitative Summary: The authors produced a comprehensive table (Table 1) mapping the specific strengths and challenges of both modalities. For example, search's biggest challenge is "Irrelevant Results and Overwhelming Information," while chat's biggest hurdle is "Inadequate for Quick Searches/General Info."

Limitations and Future Work

  • Sample Size and Diversity: The study relied on a relatively small group (130) of US-based crowdworkers. The authors acknowledge that usage patterns may differ across different cultures and languages.
  • Lack of "Transmodal" Examples: Because current commercial systems are mostly "multimodal" (offering both but keeping them separate), the authors had to rely on user descriptions of "ad hoc workflows" rather than observing true transmodal system usage.
  • Future Directions: The authors call for new "routing mechanisms" that can automatically suggest the best modality for a specific step in a task. They also envision expanding panmodality to include audio, vision, touch (tactition), and eye-gaze tracking.

Summary updated

Abstract

The emergence of generative artificial intelligence (GenAI) is transforming information interaction. For decades, search engines such as Google and Bing have been the primary means of locating relevant information for the general population. They have provided search results in the same standard format (the so-called "10 blue links"). The recent ability to chat via natural language with AI-based agents and have GenAI automatically synthesize answers in real-time (grounded in top-ranked results) is changing how people interact with and consume information at massive scale. These two information interaction modalities (traditional search and AI-powered chat) coexist in current search engines, either loosely coupled (e.g., as separate options/tabs) or tightly coupled (e.g., integrated as a chat answer embedded directly within a traditional search result page). We believe that the existence of these two different modalities, and potentially many others, is creating an opportunity to re-imagine the search experience, capitalize on the strengths of many modalities, and develop systems and strategies to support seamless flow between them. We refer to these as panmodal experiences. Unlike monomodal experiences, where only one modality is available and/or used for the task at hand, panmodal experiences make multiple modalities available to users (multimodal), directly support transitions between modalities (crossmodal), and seamlessly combine modalities to tailor task assistance (transmodal). While our focus is search and chat, with learnings from insights from a survey of over 100 individuals who have recently performed common tasks on these two modalities, we also present a more general vision for the future of information interaction using multiple modalities and the emergent capabilities of GenAI.
BibTeX
@article{white2025panmodal,
  author = {Ryen W. White and Chirag Shah},
  title = {{Panmodal Information Interaction}},
  journal = {Communications of the ACM},
  year = {2025},
  url = {https://openreview.net/forum?id=1awtFnX48W},
}

Loading related content...

Figure