Abstract
Even the most capable AI models suffer from a "collaboration gap": they perform exceptionally well solo but struggle significantly when required to work in a team. This paper introduces a novel collaborative maze-solving benchmark to quantify this phenomenon and proposes "relay inference"—using a stronger model to "seed" a conversation—as a highly effective strategy to boost the performance of weaker agents.
Introduction
As AI development shifts from monolithic models to multi-agent systems, the success of these systems depends on how well independent agents can collaborate. While benchmarks for solo reasoning are abundant, there is a lack of scalable ways to measure dynamic, natural language collaboration under partial observability. Current multi-agent solutions often rely on rigid, pre-defined protocols. However, real-world integration requires agents to communicate on the fly, establish "grounding" (mutual understanding), and resolve conflicts. This research investigates whether current training, which focuses on making AI a "helpful assistant" to humans, prepares agents to be effective teammates with other AIs.
Methodology: The Collaborative Maze-Solving Benchmark
The researchers developed a Collaborative Maze-Solving Benchmark designed to be simple yet challenging for current language models (LMs). In this setup, a maze is generated and split into two copies: Agent A sees half the cells, and Agent B sees the other half. To find a path from start to goal, they must communicate using free-form natural language dialogue.
Because the output is unstructured text, a third "grader" model (GPT-4o) extracts the agreed-upon moves and verifies them against the ground-truth maze. The study also introduces "relay inference," where a "strong" model handles the initial turns of grounding before handing the task to a "weaker" model to finish execution.
Experimental Setup
The study evaluates 32 leading open- and closed-source models from providers including OpenAI, Google, Anthropic, xAI, Meta, and DeepSeek. Core experiments utilized 6×6 mazes with 30% wall density. Ablation studies were also conducted on larger mazes up to 18×18 to observe how collaborative skills scale with task complexity.
Key Results and Insights
The Reality of the Collaboration Gap
Models that are nearly perfect solo (e.g., GPT-4.1-mini) often see their performance plummet when paired with an identical copy of themselves. This performance drop in homogeneous pairings suggests that solo reasoning capability does not automatically translate to effective teamwork.
Grounding as a Critical Skill
High-performing collaborators, such as OpenAI’s o3 or xAI's Grok-3, prioritize "grounding" in their initial turns. They establish a coordinate system (e.g., "row 0/col 0 = top-left") and share their partial map data before attempting moves. In contrast, weaker models often dive straight into local exploration without a shared plan, leading to coordination failures.
Relay Inference and Model Dynamics
Priming a dialogue with just one message from a strong model (o3) significantly boosted the performance of weaker models like Gemini-2.5-flash-lite. The strong model "seeds" the grounding, and the weaker model follows the established protocol. Interestingly, the researchers found that the order of speakers matters: outcomes are significantly better if the stronger agent speaks first. If a weaker agent starts with a poor strategy, the stronger agent often struggles to recover the dialogue.
Discussion and Future Work
The results suggest that "collaboration" is a distinct axis of capability. The "gap" is most severe in distilled models, which may retain knowledge but lose the latent skills required for coordination. Furthermore, some models exhibit "sycophancy," agreeing with incorrect partners, whereas models like Grok-3-mini were noted for being less agreeable and more corrective.
While mazes represent a lower bound for collaboration, real-world tasks involving negotiation and complex tool use likely harbor even wider gaps. Future research should move away from training models solely as "assistants" and toward training them as "collaborators" who can proactively establish shared mental models.
Summary updated