First Steps Towards Overhearing LLM Agents: A Case Study With Dungeons & Dragons Gameplay

TL;DR

This paper introduces "overhearing agents," a novel interaction paradigm where LLMs passively listen to human-to-human conversations and execute background tasks via tool calling rather than acting as direct chatbots. Using Dungeons & Dragons (D&D) as a complex, real-time testbed, the authors demonstrate that large multimodal audio models can leverage implicit vocal cues—like tone and pauses—to effectively assist a human user without disrupting the flow of social interaction.

What Problem Does This Paper Address?

Most current AI assistants are designed as "conversational agents"—they operate like chatbots, requiring direct, explicit interaction from the user. However, in many real-world scenarios (such as a business meeting, a medical consultation, or a collaborative game), stopping to explicitly type or speak to an AI is disruptive, impractical, or socially awkward.

To bridge this gap, the authors propose overhearing agents. These agents passively "listen in" on multi-party human conversations, establish beliefs about the users' underlying intentions, and execute helpful background tasks or surface suggestions silently via tool calling. To evaluate this challenging paradigm, the researchers use Dungeons & Dragons (D&D), a tabletop role-playing game characterized by long-form, spoken, and highly unpredictable conversational gameplay. Assisting the game's narrator—the Dungeon Master (DM)—presents a perfect high-cognitive-load environment for testing passive AI assistance.

Expand

Methods

The authors developed a real-time system that processes live gameplay audio in 10-second intervals, maintaining up to 15 minutes of conversational context. They evaluated five audio-enabled multimodal LLMs (GPT-4o, GPT-4o-mini, Ultravox v0.5, Qwen2.5-Omni, and Phi-4-multimodal-instruct) across two primary modalities:

1. Audio Modality: Feeding raw audio intervals directly into multimodal audio-language models.
2. Text Modality: Transcribing the audio first using OpenAI's Whisper (v3-turbo) and passing the text intervals to the LLM.

Models were prompted using a ReAct-style framework (or native function calling for OpenAI models) to enforce a hidden "chain of thought" reasoning step before invoking specific tools. The tools allowed the agent to perform three core tasks:

• Game Data Retrieval: Searching and displaying complex rules, spells, or character abilities.
• NPC Stage Director: Automatically managing non-player character (NPC) portraits on a virtual tabletop and displaying their spoken dialogue.
• Generate NPCs: Procedurally generating new characters on the fly when the DM needs to improvise.

Datasets

The authors recorded a completely new dataset consisting of 41.5 hours of novel D&D gameplay across 7 human participants, yielding 14,939 conversational turns.

To rigorously evaluate the models, they established a gold-standard benchmark of 940 human-labeled suggestions using a two-stage human annotation process:

• Stopwatch Annotation: DMs used a stopwatch during live gameplay to record exact timestamps when they would have appreciated AI assistance.
• Post-hoc Annotation: Players reviewed model-generated suggestions post-game, rating their helpfulness on a scale from -2 to +2 to establish precise precision metrics.

Key Insights and Ideas

• The "Conversational Default" Trap: Text-only models frequently suffer from a misunderstanding of the overhearing paradigm. Because instruction-tuning datasets are overwhelmingly text-based chatbots, text models often try to talk back directly to the players (e.g., generating NPC dialogue to converse with the players) rather than silently executing background tools.
• Audio Cues Unlocked Emergent Capabilities: Direct audio models (like GPT-4o) significantly outperformed text-based pipelines on complex tasks like stage direction and NPC generation. Audio inputs allow the model to perform speaker diarization and pick up on subtle acoustic cues—such as dramatic accents, emotional tone, or hesitant pauses ("Um...")—which are completely lost in plain text transcripts.
• Reasoning is Non-Negotiable: Attempting to bypass the "chain of thought" reasoning step to save on inference time resulted in a catastrophic performance drop (averaging a 70% reduction in F1 score). Overhearing agents must actively reason about short-term context to maintain an accurate internal belief of long-term conversational goals.
• Small Models Catastrophically Degenerate: Smaller open-weight models (under 10B parameters) struggled heavily with long-context multi-turn audio interactions, frequently falling into repetitive loops or outputting nonsensical suggestions over time.

Key Results

• GPT-4o (Audio) achieved the highest overall performance, demonstrating a clear emergent ability to utilize implicit audio cues for intent prediction.
• Text-based models generated massive amounts of false positives. For example, on the NPC Stage Director task, text-based GPT-4o generated 1,154 false positives (0.091 precision), whereas audio-based GPT-4o dropped to 199 false positives while boosting precision to 0.350.
• Inference Speed: Commercial models like GPT-4o managed to process 10-second audio chunks near or faster than real-time while maintaining full reasoning steps, making them viable for live deployment.

Expand

Limitations and Future Work

• Long-Context Stability: Smaller open-weight models are currently unviable for long-form overhearing tasks due to context-window degeneration, indicating a strong need for targeted distillation or specialized instruction tuning.
• Privacy and Ethics: Because overhearing agents require an "always-listening" setup, they introduce significant privacy, consent, and legal hurdles if deployed outside of closed, fully consented environments.
• Expanding Modalities: The authors plan to explore integrating visual reasoning (e.g., tracking character tokens on digital maps) and expanding the agent's toolkit to automate background music selection and sound effects.