Robots that learn their behavior in simulation usually assume their eyes run on a fixed schedule: every few milliseconds the camera delivers a fresh, fully processed list of the objects around them. In reality, the perception software that turns raw pixels into that list ends with a classical sorting-and-filtering routine — non-maximum suppression (NMS), which prunes thousands of overlapping candidate bounding boxes down to the few real ones — and its running time grows with the square of the number of candidates. A family of attacks called sponge attacks exploits exactly this: they perturb the image so the detector emits thousands of high-confidence, mutually non-overlapping "ghost" boxes that NMS must pairwise compare. On the edge computers that robots actually carry, this is not hypothetical: Phantom Sponges: Exploiting Non-Maximum Suppression to Attack Deep Object Detectors reports a 251% processing-time increase in video streams, dropping a driving pipeline from roughly 40 to roughly 16 frames per second, and Overload: Latency Attacks on Object Detection for Edge Devices reports an 11–13× single-frame slowdown on an NVIDIA Jetson with over 20,000 ghost objects per image. Meanwhile, the sim-to-real literature that trains robots to survive the physical world randomizes friction, mass, textures, and motor delay during training — but treats perception timing as either a fixed constant or mild stationary noise.
This proposal treats the attacker's schedule as one more physical property to randomize — adversarially. We will train closed-loop vision-based policies in simulation against a learned adversary that injects latency spikes calibrated to measured NMS-attack profiles from real edge hardware, so the policy repeatedly experiences the worst-case timing an attacker can arrange, and then transfer those policies zero-shot to physical hardware. The core empirical question is sharp: do policies trained under the stationary delay models used today (a fixed 15–19 ms loop as in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, or Gaussian 4–16 step delays as in Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization) collapse when delay becomes non-stationary and chosen by an adversary, and can adversarial latency randomization recover most of the task performance at modest cost on clean execution?
What Is Already Known, and What It Leaves Open
Three lines of work define the terrain. The attack literature supplies the threat model: Phantom Sponges and Beyond PhantomSponges: Enhancing Sponge Attack on Object Detection Models build universal perturbations that inflate NMS work by up to 550% while keeping real detections intact; Detector Collapse: Backdooring Object Detection to Catastrophic Overload or Blindness in the Physical World shows the effect can even be triggered by an ordinary physical object, with a triggering success rate above 75%. The evaluation literature constrains the threat model: Latency NMS Attacks: Is It Real Life or Is It Just Fantasy? shows these attacks largely evaporate on server GPUs, under INT8 quantization, and under a candidate cap, with Overload succeeding in only 2 of 6 deployment categories — so any honest proposal must restrict its claims to the edge regime where the measured slowdowns are real. The sim-to-real literature supplies the method: (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image establishes that randomizing appearance during training forces policies to ignore brittle surface detail; Isaac Gym: High Performance GPU Based Physics simulation For Robot Learning makes massively parallel training feasible; and Asymmetric Actor Critic for Image-Based Robot Learning shows a critic can use privileged simulator state while the deployed actor sees only images.
The closest existing work at the intersection is Embodied Adversarial Attack: A Dynamic Robust Physical Attack in Autonomous Driving, which trains a PPO adversary in CARLA to steer a laser projection into a moving vehicle's camera, transferring to the real world with 44–68% attack success. But that adversary corrupts what the victim perceives (misclassification); it never touches when the victim perceives it. Conversely, DT-CORL studies when but only with synthetic, stationary delay distributions on low-dimensional proprioceptive states, never an adversarial, scene-dependent spike delivered through a vision pipeline. And Can't Slow Me Down: Learning Robust and Hardware-Adaptive Object Detectors against Latency Attacks for Edge Devices defends the detector (restoring a Jetson Orin NX from 13 to 43 FPS) but never asks what the downstream controller does when latency nevertheless gets through. Nobody has connected the timing channel of post-processing attacks to closed-loop policy training — that is the gap this proposal fills, and it also resolves a real tension in the sim-to-real literature, where one school argues for exact latency identification (the 15–19 ms loop of the quadruped work; similarly Intersection-free Robot Manipulation argues precise modeling removes the need for randomization) while DT-CORL shows exact modeling fails once delays are stochastic. An adversarially scheduled spike is the extreme of that stochasticity, and neither school covers it.
Core Claim
We claim that delay robustness in closed-loop control is a property of the joint distribution of delays over an episode, not just its marginal, and that training against a budget-constrained adversary is the right way to cover the worst case. Formally, we train
$$\max_\pi \; \min_{\omega \in \Omega_B} \; J(\pi, \omega), \qquad \Omega_B = \Big\{\omega : \textstyle\sum_t \ell_t(\omega) \le B\Big\},$$
where $\pi$ is the robot's control policy, $\omega$ is a latency schedule that at each control step $t$ adds a delay $\ell_t$ drawn from a measured attack-latency library (so the policy receives a stale or missing observation rather than the current one), $B$ caps the attacker's total injected delay per episode, and $J$ is expected task return. The adversary is itself a learned policy that chooses when to spend its budget — spiking, for instance, exactly at moments where the robot commits to an irreversible maneuver. The controller-side counterpart is a policy that consumes the most recent valid observation together with a staleness signal, optionally paired with a history-based belief model in the DT-CORL style and a privileged critic in the asymmetric actor-critic style.
Research Program
Three components will be built. First, a latency threat profiler: we replay existing attacks (Phantom Sponges and Overload perturbations, generated on COCO 2017 and BDD100K frames) against YOLOv5s and YOLOv8n running with production NMS on a Jetson Orin NX, following the EVADE protocol of Latency NMS Attacks, and record per-frame end-to-end latency distributions under clean, attacked, capped, and quantized configurations. These measured distributions — not invented numbers — become the library the adversary samples from. Second, the adversarial latency scheduler: a PPO agent, in the spirit of ELA but attacking the timing channel instead of pixel content, chooses spike timings under budget $B$; inside the simulator it is implemented as a cheap table lookup, so training remains fast on Isaac Gym-class throughput. Third, delay-robust closed-loop training of the robot policy under this adversary. A small secondary thread tests the converse question raised by (CAD)$^2$RL: whether a detector backbone trained with heavy appearance randomization or style-hallucinated consistency, using the SHADE codebase, inherently resists universal sponge perturbations.
Baselines
The naive reference point is a policy trained with a fixed nominal control-loop latency — the explicit-identification practice of Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, which hard-codes a measured 15–19 ms loop — and evaluated under attack; we expect near-total collapse, mirroring how the authors of Can't Slow Me Down describe the cascading failure when frame rates drop from 43 to 13 FPS. The strongest existing competitor is DT-CORL, the only published method that trains policies explicitly robust to observation and actuation delay; it reports a normalized return of 83.3% on AntMaze umaze under 4-step deterministic delay (versus 72.3% for Augmented-CQL and 0.0% for DBPT-SAC) and roughly 105 on the delayed Adroit Hammer task where its baselines score near 0.2. To make the comparison fair, we will retrain DT-CORL on our tasks with its delay distribution widened to match the marginal statistics of our measured attack library, give it the same observation history length, the same training data, the same GPU compute budget, and the same tuning budget (an equal number of random-search configurations) as our method. Beating it means higher task success under adversarially scheduled and adaptively re-attacked latency, while staying within noise of it on benign and stationary delays. A third baseline is the perception-side defense Underload, which restores a Jetson Orin NX from 13 to 43 FPS under attack; we will run it alone, our method alone, and both stacked, to test whether control-side robustness substitutes for or complements detector hardening. Finally, the ablation that isolates our key ingredient replaces the learned adversary with independent, identically distributed latency samples drawn from the identical marginal library — same expected injected delay, no strategic timing.
Why not just…?
Why not just cap the NMS candidate queue or switch to an NMS-free detector? Latency NMS Attacks confirms that a standard max_detection = 300 cap nullifies the latency spike, and Can't Slow Me Down notes that DETR-style detectors are immune because they lack NMS. But the cap converts a slow pipeline into a blind one: thousands of confident ghosts fill the queue and real objects are dropped, the failure Detector Collapse demonstrates by collapsing Faster R-CNN mAP on VOC from 46.6% to 0.6%; and DETR-class models do not meet real-time budgets on Jetson-class accelerators, which is why YOLO variants dominate edge deployment. Crucially, our controller-side training covers both failure shapes, because a capped queue and a slow queue both reach the policy as stale or missing observations.
Why not just pre-process the input (resize, JPEG)? This works against today's digital perturbations — Beyond PhantomSponges reports dynamic resizing cuts NMS execution time by 90.18%, and Latency NMS Attacks shows JPEG compression reduces attack boxes to single digits. But Detector Collapse shows semantic physical triggers survive these filters with over 75% success, so pre-processing is not a settled defense. More fundamentally, pre-processing is a perception-side fix and our question is orthogonal and complementary: what should the controller do when latency or blindness happens anyway?
Why not just widen the delay distribution in existing delay-robust RL? This is the most serious alternative, and we treat it as such: DT-CORL already shows that jointly training a belief model and policy over randomized delay distributions beats static compensation. If sampling latency independently from the attack-shaped marginal works as well as our adversarial scheduler, the minimax machinery is unnecessary — and our matched-marginal ablation is designed precisely to detect this, in which case we will report the simpler recipe as the contribution. We still expect a gap, because an adversary concentrates delay at the moments where it is most damaging (irreversible commitments, contact transitions), a property no independent sampler captures; but the experiment, not the argument, settles it.
Why not just use high-level behavior primitives? Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks achieves 90% average success in robosuite and 93% on real Stacking with primitives that terminate on contact events rather than clock ticks. But it requires an atomic low-level fallback exactly where timing matters most (contact-rich alignment in Peg Insertion), and mobile embodiments cannot pause: a quadruped galloping at 1.18 m/s in Sim-to-Real Locomotion or a drone in flight cannot wait for a late frame. Primitives reduce exposure to timing failures; they do not remove it.
Proposed Experiments
Phase one is the threat profiler on a Jetson Orin NX, described above, using COCO 2017 and BDD100K frames; its deliverable is a latency library and an honest statement of which deployment configurations are actually vulnerable, since Latency NMS Attacks found top attacks fail in 5 of 6 deployment categories and we will not claim otherwise for the regimes where they fail. Phase two is closed-loop simulation on two platforms: CARLA driving, the same simulator ELA used for its moving-camera adversary, and a vision-based drone navigation task in the style of (CAD)$^2$RL with randomized textures and lighting. A low-dimensional sanity check on D4RL AntMaze with adversarially scheduled rather than stationary delays connects directly to DT-CORL's published 83.3% setting. Evaluation uses task success rate, collision rate, and time-to-goal under four conditions: clean, fixed attack schedules, adversarially scheduled attacks, and adaptive re-attack by a freshly trained adversary against the frozen policy — the last being the only evaluation a minimax claim can rest on. Phase three is zero-shot hardware transfer: a Jetson Orin NX running YOLOv5s with production NMS feeds a small drone or ground robot on an indoor course; the attack is injected into the live camera stream using published Phantom Sponge and Overload perturbations, and, as a stretch goal, a printed physical patch optimized with pose and lighting randomization (the diversity-over-fidelity lesson of (CAD)$^2$RL) is tested for whether it can trigger sponge behavior at all — an open question every attack paper in the file explicitly defers to future work. The decisive checks are four: stationary-trained policies should collapse under adversarial schedules while adversarially trained ones retain most clean performance; the matched-marginal ablation should show whether strategic timing matters beyond distribution shape; the Underload stacking experiment should show complementarity rather than redundancy; and if the matched-marginal ablation shows no gap, we will say so and ship the simpler recipe, which would still be the first demonstration that attack-calibrated delay randomization — rather than fixed or Gaussian delay — is what closes this particular gap.
References
- Phantom Sponges: Exploiting Non-Maximum Suppression to Attack Deep Object Detectors
- Overload: Latency Attacks on Object Detection for Edge Devices
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization
- Beyond PhantomSponges: Enhancing Sponge Attack on Object Detection Models
- Detector Collapse: Backdooring Object Detection to Catastrophic Overload or Blindness in the Physical World
- Latency NMS Attacks: Is It Real Life or Is It Just Fantasy?
- CAD2RL: Real Single-Image Flight Without a Single Real Image
- Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning
- Asymmetric Actor Critic for Image-Based Robot Learning
- Embodied Laser Attack: Leveraging Scene Priors to Achieve Agent-based Robust Non-contact Attacks
- Can't Slow Me Down: Learning Robust and Hardware-Adaptive Object Detectors against Latency Attacks for Edge Devices
- Intersection-Free Robot Manipulation With Soft-Rigid Coupled Incremental Potential Contact
- Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks