TL;DR
Researchers have developed a framework for program-level adversarial co-adaptive self-play, where AI coding agents compete to attack and defend robotic systems by writing and revising executable code. Through eight generations of evolution in driving and humanoid simulations, the study shows that while agents can develop clever software-level countermeasures, sustained robustness remains elusive as opponents quickly learn to bypass new defenses.
What Problem Does This Paper Address?
Robotic security is uniquely challenging because it bridges the gap between digital vulnerabilities and physical consequences. A simple software exploit—like the unauthenticated DDS interface recently found in the Unitree Go2—can lead to a physical crash.
Current security testing is often static. However, in the real world, attackers adapt. The authors address a critical system-level question: Can autonomous agents effectively harden a robotic stack as an opponent simultaneously evolves its attacks? Instead of just finding bugs, this paper explores the "cat-and-mouse" game of continuous software adaptation in complex environments like autonomous driving and humanoid locomotion.
Methods
The authors present a framework where "Red Teams" (attackers) and "Blue Teams" (defenders) iterate on executable code without any model fine-tuning.
Key components of their approach include:
- Program-Level Adaptation: Agents (using Claude models) modify actual Python/C++ code or middleware configurations rather than adjusting neural network weights.
- The Self-Play Loop: In each "generation," agents develop code in private workspaces. The code is then "frozen" and executed in a simulator.
- Mission-Grounded Feedback: An independent "referee" judges the outcome based on ground truth (e.g., did the car crash? did the robot fall?). This feedback, along with the opponent's code from the previous round, is fed back to the agents to inform the next version.
- Environment-Specific Interfaces: For driving, agents interact via ROS 2; for humanoids, they use DDS messages and SDK-level hooks.
Datasets and Platforms
The study evaluates two distinct robotic configurations:
- Driving: Autoware (an open-source self-driving stack) running in the CARLA simulator on a 60m clear-road route.
- Humanoid: A Unitree G1 humanoid using a walking policy and navigation stack in the MuJoCo physics engine on a 3m flat-ground path.
The coding harness utilized Claude Code (Opus for driving, Sonnet for humanoid) with specific time budgets per session (40–60 minutes).
Key Insights and Ideas
- Defensive Bottlenecks: Defenders often found success by implementing "recovery drivers" at shared actuation interfaces. By monitoring the final command sent to the motors, the Blue Team could override suspicious or conflicting instructions.
- "Boiling the Frog" Attacks: Attackers learned to evade simple threshold filters. For example, if a defender filtered out large, sudden IMU (Inertial Measurement Unit) spikes, the attacker would switch to a "gradual ramp" attack that stayed just below the detection threshold until the robot became unstable.
- The Orchestrator's Role: In the humanoid study, an "orchestrator" agent proposed different attack roles (e.g., timing attacks, sensor spoofing). This forced the defender to handle multiple types of interference simultaneously, though the defender often struggled to maintain a "general" defense that worked against all roles at once.
- Observable Artifacts: Unlike "black-box" adversarial ML, this framework produces executable code. Researchers can see exactly how the defender tried to validate state or how the attacker tried to desync sensors.
Key Results
The study tracked evolution over eight generations, revealing a volatile landscape of wins and losses.
- Driving Results: Out of 40 matches, 18 reached the goal, 16 timed out, and 6 ended in collisions. In Generation 3, the defender introduced a recovery driver that achieved a 100% success rate for that round. However, by Generation 4, the "Control" attacker learned to detect this defender and deliberately triggered a "stand-down" state, leading to a collision.
- Humanoid Results: Across 176 evaluations, only 51 reached the goal. While all "clean" (no attack) runs succeeded, 75% of attack-only runs successfully disrupted the mission.
- Recovery Scores: The humanoid defender showed "local" recovery—fixing a specific vulnerability in one generation—but often failed when the attacker shifted strategies (e.g., moving from IMU spoofing to "goal spoofing").
Limitations and Future Work
- Scenario Robustness: The study used a single eight-generation lineage for each task. To prove broad robustness, the agents would need to be tested across a wider variety of routes and environmental conditions.
- Catastrophic Forgetting: The authors noted that defenders might optimize for the current attack but inadvertently become vulnerable to an older attack they previously defeated. Future work suggests using "historical-opponent regression" to ensure new patches don't break old ones.
- Metric Sensitivity: The current "quality" scores rely heavily on peak speed and progress. These are lightweight proxies that might not capture more subtle safety or smoothness issues.
- Defender Orchestration: While the attackers had an orchestrator to distribute roles, the defender was a single agent. A "Defender Team" of specialists (e.g., one for perception, one for control) might produce more robust software.
Summary updated