Challenges in Human-Agent Communication

Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. WeldarXiv 2024 · DOI · arXiv

Abstract

As AI agents evolve from simple chatbots into autonomous systems capable of browsing the web, executing code, and managing digital workflows, the "communication gap" between humans and machines is widening. This paper identifies 12 critical challenges in human-agent communication—ranging from how agents should explain their plans to how they should learn from feedback—and provides a roadmap for building more transparent, controllable, and trustworthy AI collaborators.

Introduction

We are entering the era of "agentic" AI. Unlike traditional AI models that simply classify images or generate text, modern agents (powered by Large Language Models like GPT-4) can perceive their environment, use tools, and take actions to solve complex, multi-step problems.

However, this increased power comes with increased risk. An agent tasked with a high-level goal might take unintended actions, such as signing a user up for a new credit card to secure a flight deal or accidentally deleting critical files during a code fix. The core problem is a lack of common ground: a shared understanding between the human and the agent regarding goals, capabilities, and the state of the world. Without effective two-way communication to establish this ground, users cannot effectively monitor, verify, or steer their AI agents.

Framework and Methodology

This work synthesizes decades of research from psychology, cognitive science, and Human-Computer Interaction (HCI)—specifically drawing on Clark & Brennan’s "grounding in communication" theory—to analyze the unique hurdles posed by modern generative foundation models. By drawing on experiences building complex agentic systems like Microsoft’s AutoGen and Magentic-One, the authors categorize the communication landscape into 12 distinct challenges mapped across three interaction phases: Before, During, and After execution.

A taxonomy of 12 challenges in human-agent communication, categorized by information flow (User-to-Agent, Agent-to-User, and General) and mapped across the interaction timeline.

Key Insights into Agentic Interaction

The Shift from Classifiers to Agents

Traditional AI was often a "one-shot" interaction (e.g., "Is this a cat?"). Agents involve long-horizon, multi-step workflows where the "process" is just as important as the "result." This shift necessitates a move from simple output verification to continuous process monitoring.

The Stochasticity Paradox

Even with deterministic settings, agents operating in dynamic environments (like the live web) can produce different sequences of actions for the same prompt. This makes it incredibly difficult for a user to build a reliable "mental model" of how the agent behaves (Challenge X2).

The "Level of Detail" Tightrope

There is a fundamental tension between transparency and cognitive load. If an agent reports every single sub-step, the user is overwhelmed (Challenge X3); if it reports nothing, the user loses control. Achieving the right balance is critical for usability.

Operational Alignment vs. Cognitive Similarity

The authors clarify that "common ground" in AI doesn't mean the agent "understands" like a human. Instead, it refers to operational alignment—ensuring both parties have the same inferences about the current state and future outcomes.

Taxonomy of 12 Challenges

The paper’s primary contribution is a structured taxonomy of communication hurdles:

  1. General Challenges (X1-X4): These include the verification of behavior, maintaining consistency across runs, choosing the right level of detail, and managing long-term interaction context.
  2. User to Agent (U1-U3): These focus on communicating high-level goals, specifying nuanced preferences (e.g., "minimize cost vs. minimize time"), and providing feedback for future improvement.
  3. Agent to User (A1-A5): These involve disclosing capabilities/limitations, explaining intended plans, providing real-time status updates, reporting environmental side effects (Challenge A4), and confirming goal achievement.

Future Directions

The authors issue an "urgent call to action" for AI researchers and HCI designers to collaborate on building systems that prioritize "common ground" as a first-class citizen in agent architecture. Key areas for future work include:

  • New Design Patterns: Moving beyond "chat bubbles" to more sophisticated interfaces for plan inspection and state visualization.
  • Agent "Cards": Developing dynamic "agent cards" (similar to model cards) that show what an agent can and cannot do in a specific context.
  • Addressing Side Effects: Developing better ways for agents to communicate "latent" changes to the environment that aren't immediately obvious in a chat interface.

Summary updated

Abstract

Remarkable advancements in modern generative foundation models have enabled the development of sophisticated and highly capable autonomous agents that can observe their environment, invoke tools, and communicate with other agents to solve problems. Although such agents can communicate with users through natural language, their complexity and wide-ranging failure modes present novel challenges for human-AI interaction. Building on prior research and informed by a communication grounding perspective, we contribute to the study of \emph{human-agent communication} by identifying and analyzing twelve key communication challenges that these systems pose. These include challenges in conveying information from the agent to the user, challenges in enabling the user to convey information to the agent, and overarching challenges that need to be considered across all human-agent communication. We illustrate each challenge through concrete examples and identify open directions of research. Our findings provide insights into critical gaps in human-agent communication research and serve as an urgent call for new design patterns, principles, and guidelines to support transparency and control in these systems.
BibTeX
@misc{bansal2024challenges,
  author = {Gagan Bansal and Jennifer Wortman Vaughan and Saleema Amershi and Eric Horvitz and Adam Fourney and Hussein Mozannar and Victor Dibia and Daniel S. Weld},
  title = {{Challenges in Human-Agent Communication}},
  year = {2024},
  eprint = {2412.10380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.10380},
}

Loading related content...

Figure