Policy Search in Code Space for Multi-Agent Pathfinding

Matthias Eder, Pantelis Emmanouil, Vivek Farias, Fabian Fischer, Adam Jozefiak, Tianyi Peng, Thomas Ulz, Raphael Watschinger, Fangyu Wu, Andrew Zheng

In the world of warehouse automation and robotics, coordinating hundreds of robots to move without crashing into each other—while maximizing speed—is a nightmare of a math problem. Traditionally, engineers have had two choices: spend months hand-tuning complex rules or use "black-box" Reinforcement Learning (RL) that works well but is impossible to explain or verify.

A new paper titled "Policy Search in Code Space for Multi-Agent Pathfinding" proposes a "third way." Instead of training a neural network, the authors use LLM-based coding agents to write, test, and refine actual source code for pathfinding algorithms. The result? Algorithms that beat human experts and are still fully readable by humans.

TL;DR

The authors introduce Policy Search in Code Space (PSCS), a workflow where AI coding agents iteratively design multi-agent pathfinding algorithms in source code. This approach outperformed top human-engineered solutions by up to 58.9% in industrial simulations while producing interpretable, verifiable code with zero neural inference latency at runtime.

What Problem Does This Paper Address?

The core challenge is Lifelong Multi-Agent Pathfinding (LMAPF). In a busy warehouse, tasks arrive continuously, and a fleet of robots must navigate a shared space without deadlocking or colliding.

Current methods struggle with a trade-off:

  1. Manual Engineering: Highly interpretable and safe, but labor-intensive and hard to optimize for every specific warehouse layout.
  2. Learning-based (RL): Can find high-performance policies but lacks safety guarantees, is hard to debug, and requires expensive GPU compute at runtime.

The authors wanted to see if AI agents could automate the "engineering" part—writing the actual C++ or Python code—to get the best of both worlds: the performance of automated optimization and the transparency of traditional software.

Methods

The researchers developed a human-directed loop called Policy Search in Code Space (PSCS).

The PSCS workflow: Propose, Implement, Evaluate, Diagnose.
  1. The Loop: A coding agent (like Codex Astra) is given the environment constraints and an initial "seed" algorithm. It then iteratively proposes edits, implements them in code, runs them in a simulator, and diagnoses bottlenecks from execution traces.
  2. Hierarchical Verification: To ensure safety, every candidate policy must pass a "gauntlet": it must compile, pass unit tests for collision avoidance, and then survive short-horizon simulations before being tested on long, complex runs.
  3. Zero Latency: Because the output is standard compiled code (e.g., C++), the robots don't need to "ask" an LLM what to do at every step. They just run the high-performance binary.

Datasets

The method was evaluated across two major domains:

  • League of Robot Runners (LoRR) 2024: A competitive benchmark featuring five grid-based maps with up to 800 robots.
  • Industrial Warehouse (KNAPP): A high-fidelity simulation of real-world warehouses with irregular lanes, continuous 3D geometry, and asynchronous robot motion. The authors tested this on six distinct layouts.
Illustrative maps from the LoRR competition.

Key Insights and Ideas

  • Code as the New Weights: Instead of optimizing floating-point numbers in a neural network, the "search" happens in the space of logic and heuristics. The "weights" of the policy are the lines of code themselves.
  • Discovery of Novel Heuristics: The AI didn't just tweak numbers; it invented new strategies. For example, it discovered that having robots rotate toward their next move before a path opens up significantly reduces delays in crowded grids.
  • Provable Safety: Because the output is symbolic code, the authors were able to prove a theorem (Theorem 1) showing that their synthesized CGAR algorithm is guaranteed to be collision-free and deadlock-free under specific assumptions—something nearly impossible to do with a deep neural network.
  • Zero-Shot Generalization: A policy designed for one warehouse layout worked immediately on a completely unseen layout without any parameter tuning, matching the efficiency of the original.

Key Results

The results were starkly better than state-of-the-art human designs:

  • LoRR Benchmark: The synthesized policy, PLATM, outperformed the top competition entries by 35.0% on the densest, most congested maps.
  • Industrial Warehouse: The synthesized CGAR algorithm outperformed a calibrated, proprietary industrial solution by 58.9% in terms of orders completed per hour.
Throughput progression: the AI-designed policies (blue) quickly surpass human-engineered baselines (gray).

Limitations and Future Work

Despite the success, the authors noted several hurdles:

  • Cost: The search isn't cheap. Synthesizing the warehouse policy took about 178 hours and ~$4,800 in LLM API fees.
  • Simulation Bottleneck: The speed of the search is limited by how fast the simulator can evaluate a new piece of code.
  • Code Complexity: The AI tends to write "monolithic" code that is highly optimized but messy. Future work could involve teaching the agent to prioritize modularity and "clean code" practices.
  • Human-in-the-Loop: The current process still requires a human to set the initial objectives and occasionally steer the agent. Moving toward a fully autonomous "hardcoded search harness" is the next frontier.

Summary updated

Abstract

Developing coordination policies for complex multi-agent systems relies on either labor-intensive algorithm design or opaque task-specific reinforcement learning. With recent advances in autonomous coding agents, we show that policy search directly in code space offers a compelling third path that combines automated optimization with explicit interpretability. We investigate a human-directed workflow in which coding agents iteratively propose, implement, evaluate, diagnose, and eventually refine algorithms guided by simulation feedback and explicit correctness verification. We conducted two case studies and found that this workflow robustly outperformed any manually engineered solutions. On a challenging map in the 2024 League of Robot Runners competition, a two-day search discovered a policy that exceeds the stronger locally reproduced competition entry by 35.0% on the densest instance and generalizes to unseen map instances. In simulated real-world warehouses with irregular lanes, continuous robot geometry, and asynchronous motion, a week-long search discovered an entirely new class of multi-agent pathfinding algorithm that outperforms a calibrated proprietary solution in throughput by 58.9%. Under stated assumptions, we further prove that the new algorithm is collision-free and completes, in finite time, every accepted goal that is not withdrawn or replaced. Together, the studies show that policy search in code space powered by state-of-the-art coding agents offers a highly competitive path to building real engineering solutions that are both effective and analytically tractable.
BibTeX
@misc{eder2026policy,
  author = {Matthias Eder and Pantelis Emmanouil and Vivek Farias and Fabian Fischer and Adam Jozefiak and Tianyi Peng and Thomas Ulz and Raphael Watschinger and Fangyu Wu and Andrew Zheng},
  title = {{Policy Search in Code Space for Multi-Agent Pathfinding}},
  year = {2026},
  eprint = {2610.25946},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.25946},
}
Figure