The Collaboration Gap

Tim R. Davidson, Adam Fourney, Saleema Amershi, Robert West, Eric Horvitz, Ece Kamar

Abstract

Even the most capable AI models suffer from a "collaboration gap": they perform exceptionally well solo but struggle significantly when required to work in a team. This paper introduces a novel collaborative maze-solving benchmark to quantify this phenomenon and proposes "relay inference"—using a stronger model to "seed" a conversation—as a highly effective strategy to boost the performance of weaker agents.

Introduction

As AI development shifts from monolithic models to multi-agent systems, the success of these systems depends on how well independent agents can collaborate. While benchmarks for solo reasoning are abundant, there is a lack of scalable ways to measure dynamic, natural language collaboration under partial observability. Current multi-agent solutions often rely on rigid, pre-defined protocols. However, real-world integration requires agents to communicate on the fly, establish "grounding" (mutual understanding), and resolve conflicts. This research investigates whether current training, which focuses on making AI a "helpful assistant" to humans, prepares agents to be effective teammates with other AIs.

Methodology: The Collaborative Maze-Solving Benchmark

The researchers developed a Collaborative Maze-Solving Benchmark designed to be simple yet challenging for current language models (LMs). In this setup, a maze is generated and split into two copies: Agent A sees half the cells, and Agent B sees the other half. To find a path from start to goal, they must communicate using free-form natural language dialogue.

The Collaborative Maze-Solving Benchmark setup: a maze is split into two partially observable maps (m1, m2), requiring agents to communicate to find a path.

Because the output is unstructured text, a third "grader" model (GPT-4o) extracts the agreed-upon moves and verifies them against the ground-truth maze. The study also introduces "relay inference," where a "strong" model handles the initial turns of grounding before handing the task to a "weaker" model to finish execution.

Experimental Setup

The study evaluates 32 leading open- and closed-source models from providers including OpenAI, Google, Anthropic, xAI, Meta, and DeepSeek. Core experiments utilized 6×6 mazes with 30% wall density. Ablation studies were also conducted on larger mazes up to 18×18 to observe how collaborative skills scale with task complexity.

Key Results and Insights

The Reality of the Collaboration Gap

Models that are nearly perfect solo (e.g., GPT-4.1-mini) often see their performance plummet when paired with an identical copy of themselves. This performance drop in homogeneous pairings suggests that solo reasoning capability does not automatically translate to effective teamwork.

Grounding as a Critical Skill

High-performing collaborators, such as OpenAI’s o3 or xAI's Grok-3, prioritize "grounding" in their initial turns. They establish a coordinate system (e.g., "row 0/col 0 = top-left") and share their partial map data before attempting moves. In contrast, weaker models often dive straight into local exploration without a shared plan, leading to coordination failures.

Comparison of initial turns: o3 establishes a clear coordinate system and asks for specific data, while GPT-4.1-mini is less structured.

Relay Inference and Model Dynamics

Priming a dialogue with just one message from a strong model (o3) significantly boosted the performance of weaker models like Gemini-2.5-flash-lite. The strong model "seeds" the grounding, and the weaker model follows the established protocol. Interestingly, the researchers found that the order of speakers matters: outcomes are significantly better if the stronger agent speaks first. If a weaker agent starts with a poor strategy, the stronger agent often struggles to recover the dialogue.

Discussion and Future Work

The results suggest that "collaboration" is a distinct axis of capability. The "gap" is most severe in distilled models, which may retain knowledge but lose the latent skills required for coordination. Furthermore, some models exhibit "sycophancy," agreeing with incorrect partners, whereas models like Grok-3-mini were noted for being less agreeable and more corrective.

While mazes represent a lower bound for collaboration, real-world tasks involving negotiation and complex tool use likely harbor even wider gaps. Future research should move away from training models solely as "assistants" and toward training them as "collaborators" who can proactively establish shared mental models.

Summary updated

BibTeX
@inproceedings{davidson2026collaboration,
  author = {Tim R. Davidson and Adam Fourney and Saleema Amershi and Robert West and Eric Horvitz and Ece Kamar},
  title = {{The Collaboration Gap}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=B7TR7GNAlm},
}
Figure