When the World Is the Test Suite: Agentic Intelligence for the Physical World

Kaixin Li, Meng Luo, Yanlin Li, Yuchen Tian, Xiang Liu, Xiangru Jian, Yan Yang, Zhiheng Xi, Yang Zhang, Ziyang Luo, Mingfei Cheng, Hongzhan LinarXiv 2026 · arXiv

TL;DR

This paper explores the rise of "agentic intelligence"—coding agents that don't just plan, but write and repair the software that controls robots and industrial systems. It argues that while writing code has become cheap thanks to Large Language Models (LLMs), obtaining trustworthy evidence that the code actually works in the messy physical world remains the primary bottleneck for progress.

What Problem Does This Paper Address?

In traditional software engineering, developers rely on a "test suite"—a collection of cheap, repeatable scripts that verify if code is correct. If a test fails, the developer fixes the code and runs it again instantly.

When an AI agent writes code for a robot, however, the "test suite" is the physical world itself. Testing a program might require hours of robot time, expensive GPU-accelerated physics simulations, or human supervision. Furthermore, the "judges" (often Vision-Language Models or VLMs) used to determine if a robot succeeded are frequently wrong. The authors tackle the challenge of how agents can effectively write, test, and learn from their experiences when every "unit test" is costly, noisy, and potentially misleading.

Figure 1. Same program, different test suite. In software, the test suite usually ships with the code and returns a cheap, repeatable result. Once the program acts in the physical world, someone must build the judge, a simulator can stand in for the robot, and every physical test has to be paid for in execution.

Methods

The paper provides a comprehensive survey and framework for understanding agentic intelligence. It categorizes programs based on their "distance" from the physical actuator:

  • Action: Code that directly calls perception and control primitives (e.g., "move gripper to X, Y").
  • Orchestration: Logic that sequences pre-trained neural policies and handles error recovery.
  • Objective: Programs that define rewards or cost functions for reinforcement learning.
  • World: Code that generates the training environments, tasks, or even the robot's physical body.

The authors also analyze the "Physical CI (Continuous Integration)" loop: a cycle where an agent writes a program, runs it (in a simulator or on a robot), has the outcome judged, and then either repairs the code or adds it to a "skill library" for future use.

Figure 2. The agentic loop: Write, Run, Judge, and Repair/Store.

Datasets and Benchmarks

The paper reviews dozens of benchmarks, highlighting a shift from simple task success to "code repair" and "judge reliability." Key datasets mentioned include:

  • LIBERO-PRO: A benchmark for testing how well agents handle perturbations in objects, goals, and environments.
  • FailBench: A dataset specifically designed to evaluate how accurately VLMs can judge whether a manipulation task succeeded or failed.
  • RoboDojo & CaP-Bench: Environments used to test agents that write Python code for robot control.
  • RoboBugBench: A new type of benchmark (released Oct 2026) that focuses on repairing real-world defects in existing robotics software stacks like Nav2.

Key Insights and Ideas

  • Programs vs. Policies: Programs (explicit code) tend to outperform end-to-end neural policies when the structure of a task changes (e.g., a new sequence of steps). However, neural policies are still better at absorbing "sensorimotor noise" (the jitter and uncertainty of real-world physics).
  • The "Judge" is the Weak Link: We often assume we can use an LLM/VLM to tell us if a robot succeeded. The paper reveals that the best detectors only reach ~77% accuracy on manipulation tasks. If the judge is wrong, the agent "learns" the wrong lessons, filling its library with "successful" failures.
  • Experience Decay: Just like software, robot skill libraries suffer from "bit rot." A skill that worked yesterday might fail today if the camera moves slightly or the robot's joints loosen. The paper identifies "retention rate" as a critical, yet rarely reported, metric.
  • Cost is a First-Class Citizen: Evaluating a single reward function through policy training can cost $15 in API fees and 24 hours of GPU time. Progress requires "cheap proxies"—simulators or formal checks—that can prune bad ideas before they reach the robot.

Key Results

  • ASPIRE System: By adding an execution engine and a skill library, researchers saw macro-average success rates on manipulation benchmarks jump from 14% to 62%.
  • Judge Error: In one study (Thea), learned judges accepted 6.7% of failed and 13.3% of unfinished tasks as "successes."
  • Sim-to-Real: In the Agentic RSR study, 8 out of 10 tasks solved by an agent in a reconstructed simulation successfully transferred to a real physical robot.

Limitations and Future Work

The authors argue that current research is too focused on "writing" and not enough on "verifying." They propose several "First Experiments" for the field:

  1. Physical Continuous Integration: Developing procedures that automatically run regression tests on a robot's entire skill library whenever a single line of code is changed.
  2. Robust Judges: Creating success detectors that can't be "gamed" by an agent searching for a high score.
  3. Measuring Staleness: Systematically testing how much of a skill library survives when one factor (like lighting or a robot's grip strength) is changed.

Ultimately, the paper concludes that for robots to truly improve themselves, we must treat the physical world with the same rigor we apply to software repositories—testing the tests as much as the code.

Summary updated

Abstract

Coding agents built to repair software now write, test, and repair programs for robots, laboratories, and industrial controllers. Unlike in software engineering, no test suite comes with the code. Whether a program works must be established by running it on a robot, in a physics simulator, or through the policies it helps to train, and such evidence is costly to obtain and can itself be wrong. As these agents take on more physical tasks, how their programs are judged matters as much as how they are written. This paper examines this emerging area, agentic intelligence for the physical world, by treating the world as the program's test suite. We organize the discussion around three questions: how agents write programs for the physical world, how these programs are tested, and how agents learn from the results. We first trace how language models for robots and software engineering agents came together, ask which software testing practices carry over to physical tests, and describe each program by where it sits relative to the actuator and by what judges it. For writing, we cover programs that act as robot policies and programs that support policy learning by producing rewards, tasks, and training worlds. For testing, we examine what physical trials cost, what traces, learned judges, simulators, and formal checks can establish in their place, and what safety requires beyond task success. For learning, we discuss how tested programs are retained as experience, in skill libraries and in model weights. Finally, we turn to benchmarks and evaluation practice and outline open challenges. Writing programs is becoming cheap, while trustworthy evidence about what they do remains costly. Progress will therefore depend on a physical counterpart of continuous integration: verification cheap enough to run often, reliable enough to trust under search, and maintained alongside the experience it validates.
BibTeX
@misc{li2026when,
  author = {Kaixin Li and Meng Luo and Yanlin Li and Yuchen Tian and Xiang Liu and Xiangru Jian and Yan Yang and Zhiheng Xi and Yang Zhang and Ziyang Luo and Mingfei Cheng and Hongzhan Lin},
  title = {{When the World Is the Test Suite: Agentic Intelligence for the Physical World}},
  year = {2026},
  eprint = {2611.22291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22291},
}
Figure