Program-Level Adversarial Co-Adaptive Self-Play for Robotic Security

Leo Benaharon, Chi Zhang, Wei Zheng, Weifan Zhang, Yifan WeiarXiv 2026 · arXiv

TL;DR

Researchers have developed a framework for program-level adversarial co-adaptive self-play, where AI coding agents compete to attack and defend robotic systems by writing and revising executable code. Through eight generations of evolution in driving and humanoid simulations, the study shows that while agents can develop clever software-level countermeasures, sustained robustness remains elusive as opponents quickly learn to bypass new defenses.

What Problem Does This Paper Address?

Robotic security is uniquely challenging because it bridges the gap between digital vulnerabilities and physical consequences. A simple software exploit—like the unauthenticated DDS interface recently found in the Unitree Go2—can lead to a physical crash.

Current security testing is often static. However, in the real world, attackers adapt. The authors address a critical system-level question: Can autonomous agents effectively harden a robotic stack as an opponent simultaneously evolves its attacks? Instead of just finding bugs, this paper explores the "cat-and-mouse" game of continuous software adaptation in complex environments like autonomous driving and humanoid locomotion.

Methods

The authors present a framework where "Red Teams" (attackers) and "Blue Teams" (defenders) iterate on executable code without any model fine-tuning.

Figure 1: Program-level adversarial co-adaptation. Coding agents develop attack and defense pro- grams that are checked, frozen, and evaluated in Autoware-CARLA or G1-MuJoCo. Independent referees judge outcomes from simulator ground truth; results and tested opponent code guide the next generation.

Key components of their approach include:

  • Program-Level Adaptation: Agents (using Claude models) modify actual Python/C++ code or middleware configurations rather than adjusting neural network weights.
  • The Self-Play Loop: In each "generation," agents develop code in private workspaces. The code is then "frozen" and executed in a simulator.
  • Mission-Grounded Feedback: An independent "referee" judges the outcome based on ground truth (e.g., did the car crash? did the robot fall?). This feedback, along with the opponent's code from the previous round, is fed back to the agents to inform the next version.
  • Environment-Specific Interfaces: For driving, agents interact via ROS 2; for humanoids, they use DDS messages and SDK-level hooks.

Datasets and Platforms

The study evaluates two distinct robotic configurations:

  1. Driving: Autoware (an open-source self-driving stack) running in the CARLA simulator on a 60m clear-road route.
  2. Humanoid: A Unitree G1 humanoid using a walking policy and navigation stack in the MuJoCo physics engine on a 3m flat-ground path.

The coding harness utilized Claude Code (Opus for driving, Sonnet for humanoid) with specific time budgets per session (40–60 minutes).

Key Insights and Ideas

  • Defensive Bottlenecks: Defenders often found success by implementing "recovery drivers" at shared actuation interfaces. By monitoring the final command sent to the motors, the Blue Team could override suspicious or conflicting instructions.
  • "Boiling the Frog" Attacks: Attackers learned to evade simple threshold filters. For example, if a defender filtered out large, sudden IMU (Inertial Measurement Unit) spikes, the attacker would switch to a "gradual ramp" attack that stayed just below the detection threshold until the robot became unstable.
  • The Orchestrator's Role: In the humanoid study, an "orchestrator" agent proposed different attack roles (e.g., timing attacks, sensor spoofing). This forced the defender to handle multiple types of interference simultaneously, though the defender often struggled to maintain a "general" defense that worked against all roles at once.
  • Observable Artifacts: Unlike "black-box" adversarial ML, this framework produces executable code. Researchers can see exactly how the defender tried to validate state or how the attacker tried to desync sensors.

Key Results

The study tracked evolution over eight generations, revealing a volatile landscape of wins and losses.

Figure 2: Driving. A crash, collision, or off-road event results in an attacker win; successful com- pletion results in a defender win; and a timeout results in a tie.
  • Driving Results: Out of 40 matches, 18 reached the goal, 16 timed out, and 6 ended in collisions. In Generation 3, the defender introduced a recovery driver that achieved a 100% success rate for that round. However, by Generation 4, the "Control" attacker learned to detect this defender and deliberately triggered a "stand-down" state, leading to a collision.
  • Humanoid Results: Across 176 evaluations, only 51 reached the goal. While all "clean" (no attack) runs succeeded, 75% of attack-only runs successfully disrupted the mission.
  • Recovery Scores: The humanoid defender showed "local" recovery—fixing a specific vulnerability in one generation—but often failed when the attacker shifted strategies (e.g., moving from IMU spoofing to "goal spoofing").
Figure 5: Humanoid. Each column progresses from top to bottom. The red marker indicates the goal. (a) IMU spoofing causes a fall without defense; (b) the defender mitigates the same attack; (c) Position drift sends the robot off-route; (d) Combined sensor spoofing and motor-command injection and (e) competing actuator commands both cause falls.

Limitations and Future Work

  • Scenario Robustness: The study used a single eight-generation lineage for each task. To prove broad robustness, the agents would need to be tested across a wider variety of routes and environmental conditions.
  • Catastrophic Forgetting: The authors noted that defenders might optimize for the current attack but inadvertently become vulnerable to an older attack they previously defeated. Future work suggests using "historical-opponent regression" to ensure new patches don't break old ones.
  • Metric Sensitivity: The current "quality" scores rely heavily on peak speed and progress. These are lightweight proxies that might not capture more subtle safety or smoothness issues.
  • Defender Orchestration: While the attackers had an orchestrator to distribute roles, the defender was a single agent. A "Defender Team" of specialists (e.g., one for perception, one for control) might produce more robust software.

Summary updated

Abstract

Coding agents can adapt robotic software by revising executable programs, but successful responses to an evolving opponent may not remain effective. We present a framework for program-level adversarial co-adaptive self-play in which coding agents generate attacks and defenses from tested opponent code and mission feedback, without model fine-tuning. Environment-specific interfaces support external programs for autonomous driving and SDK-level defenses for humanoid locomotion. Two eight-generation case studies reveal concrete software responses: a driving defender introduces direct-actuation recovery, while a humanoid defender revises incoming-state validation. These responses provide local task recovery but do not establish sustained robustness. The framework makes such adaptation and limitations observable through executable artifacts and task-level evidence.
BibTeX
@misc{benaharon2026program,
  author = {Leo Benaharon and Chi Zhang and Wei Zheng and Weifan Zhang and Yifan Wei},
  title = {{Program-Level Adversarial Co-Adaptive Self-Play for Robotic Security}},
  year = {2026},
  eprint = {2610.36143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36143},
}
Figure