When the Plan Changes: A Strategic World Testbed for Dynamic Multi-Agent Coordination

Joon-Seok Kim, Eugene Agichtein, Kai Shu, Liang Zhao, Andreas ZüflearXiv 2026 · arXiv

TL;DR

This paper introduces Strategic World, a simulation testbed designed to diagnose why multi-agent systems fail when objectives change mid-stream. By strictly separating authoritative world state from local agent plans and observations, the researchers demonstrate that standard "success" metrics hide critical coordination failures, and they propose a simple "outbox guard" that significantly improves team performance by prioritizing information sharing over local work.

What Problem Does This Paper Address?

In multi-agent systems, especially those powered by Large Language Models (LLMs), "intent drift" is a major hurdle. When a mission objective changes, agents often continue executing obsolete plans, ignore new constraints, or fail to notify their teammates.

The core challenge is that current benchmarks often only look at the final outcome (did the task get done?). This "blunt instrument" approach fails to distinguish between different types of failure:

  • Uninformed work: The agent didn't receive the update.
  • Post-receipt obsolete work: The agent got the update but kept doing the old task anyway.
  • Blocked unsafe intent: The agent tried to take a forbidden shortcut but was stopped by the environment.

Without distinguishing these, we can't build agents that reliably coordinate in dynamic, high-stakes environments like disaster relief or infrastructure repair.

Methods

The authors developed Strategic World, a graph-structured simulation environment. Unlike many testbeds where agents have "god-like" access to the global state, Strategic World enforces strict information boundaries.

Figure 2: Selected capabilities and relationships in Strategic World. Purple nodes denote geopolitical capabilities, brown national strategic capabilities, blue operational capabilities, green tactical capabilities, and gray cross-cutting services. Arrows denote the labeled semantic relationships, including dependencies, authority, observation, and evaluation; they are not a chronological message-flow diagram. The coordination experiments instantiate the operational assignment–communication–execution path within this broader model.

Key technical features include:

  • State Separation: It distinguishes between the Authoritative Objective (the ground truth) and the Local Plan (what the agent thinks it should be doing).
  • Intervention Families: The testbed includes six scenarios to stress-test coordination, such as "Partition" (where an agent loses comms) and "False Authority" (where agents receive unauthorized/misleading instructions).
  • Trace-Based Diagnosis: Instead of just checking for success, the system records every action intent to identify why a failure occurred.
  • The Outbox Guard: A novel policy intervention that forces an agent to confirm it has submitted update notices to all peers before it is allowed to proceed with its own local work.
Figure 3: Overall testbed design. Each actor receives a local projection and selects action intents; the simulator commits their consequences and delivers due notices. Full state and compatibility classifications are retained for offline assessment, not policy feedback. The study uses calibration and a separate discovery trajectory to select an intervention, freezes the protocol, and evaluates matched held-out settings. Actor-local assistance is an explicit policy component, not a change to world legality.

Datasets

The paper evaluates its framework using a suite of generated scenarios within the Strategic World simulation:

  • Principal Diagnostic Suite: 180 episodes crossing six scenario families and six timing configurations.
  • Calibration Episodes: 294 episodes used to establish baseline feasibility and protocol limits.
  • LLM Pilots: Evaluated using qwen3:4b-instruct as the primary agent model.
  • Held-out Evaluation: 16 episodes used to test the "outbox guard" intervention against assisted baselines.

Key Insights and Ideas

  • The Acknowledgment-Behavior Gap: An agent saying "Understood" to a new objective does not guarantee it will change its behavior. The authors found cases where agents acknowledged a change but spent their next turns performing obsolete work.
  • Summaries are Deceptive: Identical failure summaries (e.g., "Success: False, Tasks: 0") can mask completely different root causes. One agent might be waiting for info, while another might be actively trying to violate a safety constraint.
  • Communication is a Bottleneck, not just a Tool: In dynamic settings, the timing of communication is as important as the content. Agents that act on new info before sharing it often cause team-wide failure, even if their individual actions are "correct" for the new goal.
  • Provenance Matters: The testbed explicitly tracks who authorized a change, allowing researchers to study how agents handle "forged" instructions or prompt injections.

Key Results

The most striking result came from the "outbox guard" intervention:

  • Completion Rate: The engineered guard completed 4/4 held-out settings, whereas the calibrated assisted baseline only managed 1/4.
  • Authorized Work: The guard ensured 39/39 units of authorized work were completed, compared to 33/39 for the baseline.
  • Latency: While the baseline sometimes sent a first relay earlier, it often failed to reach all peers. The guard ensured complete dissemination, reducing the "post-adoption work before submission" from 10 actions to zero.
Table 1: Scenario interventions and the coordination distinctions they expose.

Limitations and Future Work

The authors are careful to note that these results do not establish the reliability of "frontier" models (like GPT-4 or Claude 3) in these settings, as they primarily tested smaller, local models.

Limitations include: * Fictional Scenarios: The tasks (water network repair) are schematic and don't capture the full complexity of real-world physics or social dynamics. * Fixed Membership: While the testbed supports agents joining/leaving, it doesn't yet evaluate agents spontaneously forming new teams or negotiating roles from scratch. * Policy Engineering: The "outbox guard" is a piece of explicit policy engineering; the goal for future work is to see if LLMs can learn or reason their way to such strategies autonomously.

Future research will likely focus on scaling these diagnostic tools to larger agent populations and more complex, open-ended environments.

Summary updated

Abstract

Changing an objective does not automatically revise agents' earlier plans or delegated tasks. Acknowledgment can coexist with obsolete execution, while misleading instructions can redirect work or discard retained constraints. We present a Strategic World simulation testbed that separates authoritative objectives, local plans, notice receipt, and consequential action. Six intervention families vary objectives, conditions, authority, membership, and communication. Matched protocols and deadlines establish feasibility for the tested disturbances. Controlled trajectories demonstrate that identical completion and acknowledgment summaries can conceal uninformed obsolete work, obsolete work after receipt, or blocked unsafe intent. Local language-model trials and instructed action probes reveal competence limits that constrain interpretation of dynamic failures. A further 16-episode comparison evaluates an engineered guard that prioritizes pending notice submissions over local work. It completes 4/4 held-out settings, compared with 1/4 for a calibrated assisted baseline. Each episode contains one large language model actor and two fixed peers. These results establish diagnostic distinctions and bounded policy feasibility under explicit simulation assumptions; they do not establish frontier-model reliability or autonomous team adaptation.
BibTeX
@misc{kim2026when,
  author = {Joon-Seok Kim and Eugene Agichtein and Kai Shu and Liang Zhao and Andreas Züfle},
  title = {{When the Plan Changes: A Strategic World Testbed for Dynamic Multi-Agent Coordination}},
  year = {2026},
  eprint = {2611.10011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10011},
}
Figure