Correctness Boundaries of Typed Plans for Natural-Language Robot Fleet Control

Artem Tuzov, Dmitrii Vizitei, Evgenii Shlomov, Senya Belykh, Anastasia Malakhova, Alexander MaloletovarXiv 2026 · arXiv

TL;DR

This paper introduces a framework for controlling robot fleets using natural language by bridging the gap between Large Language Model (LLM) intent and physical execution. By implementing a three-level architecture with "typed plans" and deterministic guards, the authors demonstrate how to catch logical errors—like duplicate task assignments or infeasible formations—that LLMs often miss, ensuring that what the operator asks for is what the fleet actually delivers.

What Problem Does This Paper Address?

While LLMs are excellent at translating human language into high-level goals, they struggle with the precision required for multi-robot coordination. A simple command like "Move cargo from Security to Admin" involves complex sub-tasks: assigning specific robots, avoiding collisions, and managing formations.

Current systems often suffer from two main issues:

  1. Logical Inconsistency: An LLM might assign two robots to the same pickup or generate a plan that is structurally valid but semantically nonsensical.
  2. Reporting Gaps: If a low-level path planner modifies a plan for feasibility (e.g., dropping a robot that can't find a path), the system might report "Success" based on the modified plan, leaving the operator unaware that the original request was only partially fulfilled.

Methods

The authors propose a three-level system implemented in ROS 2 (Robot Operating System 2):

  1. Level 1: Language Orchestration: An LLM turns natural language into a Typed Plan (a recursive JSON structure). This plan is then passed through Deterministic Interface Checks (guards) that validate structure and apply rules, such as the "single-carrier rule" to prevent redundant assignments.
  2. Level 2: Fleet Coordination: A dedicated Multi-Agent Path Finding (MAPF) backend (using LNS2) handles the geometric realities of moving twenty robots without collisions.
  3. Level 3: Per-Robot Execution: Individual robots run Nav2 stacks in a simulated environment (Stage).
The three-level system architecture ensures that high-level language commands are filtered through deterministic checks before reaching the coordination and execution layers.

A key innovation is the use of Task Evaluators that monitor pose-based events (e.g., "Is robot X actually in the pickup zone?") independently of the robot's own status reports. This provides a ground-truth audit of whether the original command's requirements were met.

Datasets

The study utilizes two primary evaluation methods:

  • Offline Benchmark (E1): A new benchmark consisting of 44 cases across 12 categories (navigation, formation, disbanding, etc.). It tests various LLMs (Qwen, Gemma, Mistral, Gemini, Nemotron) on their ability to generate valid JSON plans under different "decoding configurations" (e.g., using a response schema vs. raw text).
  • Simulation Series (E2): An 18-run simulation series using the "Amongus" map with 20 differential-drive robots. These runs involve complex "M2" (5 tasks) and "M3" (9 tasks) missions designed to trigger edge cases like shared drop-off locations.

Key Insights and Ideas

  • Structural Validity $\neq$ Semantic Correctness: A plan can pass a JSON schema check but still be a failure. For example, the authors found cases where a plan was structurally perfect but requested a formation shape that the planner later rejected as infeasible.
  • The "Success" Illusion: Low-level components (like the MAPF planner) might report "All agents arrived," but if the planner silently removed a robot from the set during a replan, that "success" is misleading. The authors argue for reporting the requested vs. arrived sets separately.
  • Deterministic Guards are Essential: The "single-carrier guard" successfully caught and corrected redundant pickup assignments in several runs, showing that hard-coded logic is still a necessary safety net for LLM-generated plans.
  • Schema Sensitivity: Interestingly, forcing an LLM to use a response schema (Schema-on) didn't always help. For some models, like Qwen3.5-9B, the pass rate dropped significantly (from 87.3% to 19.1%) when a schema was enforced, suggesting that the constraint interfered with the model's reasoning.

Key Results

  • Offline Performance: Across local LLM configurations, 70.0–87.3% of generated plans passed structural and basic requirement checks.
  • Simulation Success: All 18 runs in the selected simulation series eventually met their pose-based task conditions, though some required the deterministic guards to fix assignment errors first.
  • Error Categorization: As shown in the paper's analysis, the most common failures in local models were related to "Formation fields or staging" and "Plan structure."
The authors categorized failures to identify exactly where the language-to-plan translation breaks down.

Limitations and Future Work

  • Scope: The study was limited to one specific simulation map and two main mission types.
  • Physicality: The simulation tracks zone presence but does not model physical cargo transfer, collision physics, or sensor noise.
  • Future Directions: The authors suggest that future systems should focus on better "traceability"—ensuring that every change made by a guard or a planner is explicitly linked back to the operator's original command. They also motivate the need for "mission supervision" that can request additional work if the initial plan fails to meet all requirements.

Summary updated

Abstract

We examine what plan validation and execution reports establish in natural-language robot fleet control. Commands are traced through typed plans, deterministic corrections, fleet coordination and outcome checks in a three-level Robot Operating System 2 implementation with twenty simulated robots. An offline benchmark of 44 cases checks plan structure and selected command requirements. Across five local configurations without a response schema, 70.0-87.3% of responses pass. All runs in a selected 18-run simulation series meet their pose-based task conditions. This series includes two successful repeats that replaced incomplete runs after a software change. Saved plans and logs document duplicate pickup assignments and goal adjustments within the original task zones. Separate formation traces show a structurally accepted plan with the wrong formation shape and a rejected route, and an arrival report that omitted a requested robot after replanning. Carry completion records pickup and subsequent drop-off zone presence for the same robot. The cases distinguish changes to executable actions from changes to the requirements covered by a completion report. They motivate checking the effective assignments, goals and reported outcome against the original command.
BibTeX
@misc{tuzov2026correctness,
  author = {Artem Tuzov and Dmitrii Vizitei and Evgenii Shlomov and Senya Belykh and Anastasia Malakhova and Alexander Maloletov},
  title = {{Correctness Boundaries of Typed Plans for Natural-Language Robot Fleet Control}},
  year = {2026},
  eprint = {2611.12654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12654},
}
Figure