Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap

Andrew Zhu, Chris Callison-BurcharXiv 2025 · arXiv

TL;DR

This paper formalizes the paradigm of "overhearing LLM agents"—AI assistants that passively monitor human-to-human interactions (via audio, text, or video) to provide contextual help without direct prompting. The authors establish a comprehensive taxonomy for these non-intrusive systems, outline key user interaction design principles, and present a research roadmap to tackle open challenges like real-time stream processing and multi-party privacy consent.

What Problem Does This Paper Address?

Most current Large Language Model (LLM) agents operate as conversational chatbots. To get help, a user must stop what they are doing, formulate a prompt, and wait for a response. While effective for isolated tasks, this direct-interaction model fails in collaborative, fast-paced human-to-human settings like a doctor consulting a patient, a teacher leading a classroom, or colleagues brainstorming around a whiteboard.

Figure 1: Conversational AI agents (top) are presented as a chatbot with which the user converses directly. We present overhearing AI agents (bottom), which instead “listen in” on human-to-human conversations, and provide suggestions to the user without needing to participate in the conversation.

The authors address the lack of a formalized framework for an alternative paradigm: overhearing agents. These are passive AI systems that "listen in" on ambient activity and intervene only when they can provide value, completely bypassing the need for explicit user prompts. This shift introduces unique challenges, primarily because the agent must infer user intent implicitly without interrupting the flow of human conversation to ask clarifying questions.

Methods

As a survey and foundational roadmap paper, the authors synthesize literature across multiagent communication, ubiquitous computing, AI copilots, and proactive dialogue systems. They organize the design space of overhearing agents into a structured taxonomy divided into two primary categories:

  1. User Interaction: How the agent perceives and communicates with humans. This includes Initiative (always active vs. user-initiated vs. rule-based), Input Modality (audio, text, video), and Interfaces (desktop, wearables, smart home).
  2. System Design: How the underlying architecture processes tasks. This includes State (read-only vs. read-write), Timeliness (real-time vs. asynchronous), and Interactivity (foreground suggestions vs. background world-modeling).

The authors also derive five core interaction design principles for developers: suggestions must be verifiable at a glance, dismissible without friction, reversible (undoable), editable, and managed by an intelligent queuing system to avoid user cognitive fatigue.

Datasets

As a survey paper, no new datasets or benchmarks were introduced. However, the authors highlight existing multimodal datasets that serve as groundwork for this paradigm, specifically pointing to EgoExo4D—a dataset containing first- and third-person video footage of skilled human activities with detailed expert annotations, which is highly relevant for video-based overhearing agents.

Key Insights and Ideas

  • "Thinking" vs. "Talking": Unlike standard chatbots, overhearing agents should never output verbose conversational text to the user. Instead, they should use their language generation capabilities internally for "thinking" (chain-of-thought reasoning) to decide if and how to invoke a tool or surface a glanceable suggestion.
  • The Initiative Trade-off: Continuous "Always Active" monitoring ensures the agent never misses an opportunity to help, but it introduces massive privacy concerns and token costs. Conversely, "User-Initiated" triggers (like a button press on a smartwatch) reduce agent activations to just 1/5th of the always-active rate, but risk missing crucial conversational context.
  • The Taxonomy Matrix: The paper provides a clean mapping of how existing tools (like coding copilots or meeting note-takers) fit into the broader overhearing ecosystem, helping developers identify unexplored design combinations.
Taxonomy of Overhearing Dimensions

Key Results

The paper's primary contribution is qualitative: the first unified taxonomy and architectural framework for overhearing LLM agents.

Quantitatively, the survey highlights critical human-computer interaction (HCI) baselines from adjacent studies to justify its design principles. For instance, it notes that less than half of surveyed populations currently feel comfortable with "always-listening" ambient AI devices, underscoring that the success of this paradigm hinges entirely on local, privacy-preserving engineering rather than just raw model accuracy.

Limitations and Future Work

The authors outline five major research roadblocks that define the roadmap for overhearing agents:

  1. Predicting Intervention Points: How can an agent accurately determine the exact millisecond to intervene in a continuous audio stream? Current models struggle with the latency of splitting audio into discrete turns.
  2. Evaluation Metrics: We lack standard metrics to measure the cognitive load of "spurious suggestions" (false positives) against the helpfulness of correct ones.
  3. Multimodal Throughput: Processing continuous video/audio streams in real time is computationally expensive. The authors suggest developing "variable-rate tokenization" schemes that compress periods of silence or stillness to save compute.
  4. Software Infrastructure: Existing agent frameworks (like LangChain or AutoGPT) are text-first and synchronous. New libraries must be designed from the ground up for native audio/video I/O and asynchronous background execution.
  5. Multi-Party Consent: In a room with multiple people, how does an agent selectively process data only from individuals who have opted in? Selective privacy-masking algorithms remain an open challenge.

Summary updated

Abstract

Imagine AI assistants that enhance conversations without interrupting them: quietly providing relevant information during a medical consultation, seamlessly preparing materials as teachers discuss lesson plans, or unobtrusively scheduling meetings as colleagues debate calendars. While modern conversational LLM agents directly assist human users with tasks through a chat interface, we study this alternative paradigm for interacting with LLM agents, which we call "overhearing agents." Rather than demanding the user's attention, overhearing agents continuously monitor ambient activity and intervene only when they can provide contextual assistance. In this paper, we present the first analysis of overhearing LLM agents as a distinct paradigm in human-AI interaction and establish a taxonomy of overhearing agent interactions and tasks grounded in a survey of works on prior LLM-powered agents and exploratory HCI studies. Based on this taxonomy, we create a list of best practices for researchers and developers building overhearing agent systems. Finally, we outline the remaining research gaps and reveal opportunities for future research in the overhearing paradigm.
BibTeX
@misc{zhu2025overhearing,
  author = {Andrew Zhu and Chris Callison-Burch},
  title = {{Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap}},
  year = {2025},
  eprint = {2509.16325},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.16325},
}
Figure