Training Robot Policies for Latency Spikes That Arrive When the Scene Gets Hard

Robots trained in simulation are usually made robust by randomizing the simulator. Each training episode draws new masses, frictions, textures, and sensor delays, so the policy never overfits to one version of the world. Delays are almost always drawn independently of what is happening in the scene, from a fixed range. Real robot software does not behave this way. Perception and planning slow down exactly when the scene is cluttered, crowded, or fast-changing, which is also when a timely reaction matters most. A policy trained under independent delay noise has never practiced the combination that causes real crashes: a hard situation seen through stale sensor data.

We propose to change two things, and to test which one matters. First, the simulator's delay model becomes state-coupled: the delay applied to each sensor stream depends on how complex the current scene is, with persistence (spikes come in bursts) and a heavy upper tail. The parameters of this coupling are randomized like any other domain parameter. Second, training optimizes the average over the worst episodes rather than the average over all episodes, and actively searches for coupling parameters that break the current policy. Our central claim is that coupling explains a large share of the damage from delay. Concretely, a policy trained under coupled delays should beat one trained under independent delays with the same distribution of delay values on tail safety metrics, and a risk-averse objective alone should not close that gap.

Background and the gap

Two lines of work sit on either side of this proposal. On the sim-to-real side, Sim-to-Real: Learning Agile Locomotion For Quadruped Robots modeled a fixed 15–19 ms control-loop latency alongside dynamics randomization and showed that this was needed for agile gaits to transfer to the Minitaur. Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization (MMDR) goes further. It randomizes the delays of depth images and body sensors separately, and roughly doubles real-world moving distance on an A1 quadruped. Its delays are still drawn uniformly and independently of the scene, and it optimizes expected return. These two papers supply our training setup, and MMDR is the method we must beat.

On the tail-latency side, COLA: Characterizing and Optimizing the Tail Latency for Safe Level-4 Autonomous Vehicle Systems motivates the proposal. It shows on Apollo and Autoware that traffic density drives bursty computation and memory contention, so tail latency is caused by the scene. COLA fixes this at the systems layer with fast-path planning and work stealing, and its fast path costs a 6.43% longer commute. FogROS2-PLR: Probabilistic Latency-Reliability For Cloud Robotics cuts P99 network latency with redundant carriers but explicitly offers only probabilistic guarantees. Neither paper changes the control policy, so we treat both as complementary to our work rather than as competitors.

The closest learning-side work is Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments (PDTS). It trains for the worst 10% of outcomes by choosing hard randomized tasks. It uses a learned predictor of task difficulty and a diversity penalty, because always picking the single hardest tasks makes the chosen tasks collapse into a narrow cluster. PDTS supplies our sampling method, but it only randomizes static physical parameters. Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization (DT-CORL) learns a Transformer that estimates the current state from a history of delayed observations. It works offline under bounded step delays of 4–16 steps and reports average return. We may reuse its belief architecture, but it does not address scene-dependent delays or tail objectives.

The obvious combination, "PDTS or CVaR over MMDR's delay ranges," is not in the file. It is, however, simple enough that a reviewer could fairly call it incremental. We therefore include it as a baseline rather than as our contribution. The new element is the coupling between scene and delay, and the experiments are built to isolate it.

Core Claim

Let $c_t$ be a scalar measure of scene complexity at time $t$, such as obstacle count in view, nearby agent density, or depth-image clutter. Let $d_t^{m}$ be the delay of sensor stream $m$. We model the delay as

$$ d_t^{m} = b^{m} + \beta^{m} c_t + s_t^{m}, \qquad s_t^{m} = \rho^{m} s_{t-1}^{m} + \epsilon_t^{m}, $$

where $b^m$ is the base latency and $\beta^m$ is the sensitivity of delay to scene complexity. The burst term $s_t^m$ carries over from step to step with persistence $\rho^m$, and its new-noise term $\epsilon_t^m$ is drawn from a heavy-tailed distribution with tail index $\kappa^m$. The vector $\xi = (b, \beta, \rho, \kappa)$ is the randomized "delay task." The policy is trained to maximize the conditional value-at-risk (CVaR), the mean return over the worst $\alpha$-fraction of episodes:

$$ \max_\pi \; \mathrm{CVaR}_\alpha\big[R(\pi, \xi, \omega)\big], $$

where $R$ is episode return and $\omega$ covers the physical parameters and scenes. Following PDTS, we use $\alpha = 0.9$ in its convention, meaning the worst 10%. The claim is falsifiable because we can build a control condition, called matched-marginal independent delays. In this condition we record every delay value produced by the coupled model, then shuffle those values across timesteps and episodes. The policy sees exactly the same delay values but loses their link to $c_t$. If coupled training does not beat matched-marginal training on tail metrics, the coupling hypothesis is wrong.

At deployment the policy also receives the measured age of each sensor stream, which hardware timestamps make available. Under coupling, a rising age is itself a warning that the scene is getting hard, and we expect the policy to learn to slow down or widen its safety margins in response. This is a learned, graded version of COLA's fast path.

Research Program

The first step is empirical and could end the project early. We will instrument the A1 perception-and-control stack, meaning the depth camera pipeline plus policy inference, and log delay per stream against scene clutter across indoor and outdoor runs. We will fit $\xi$ to these logs. COLA establishes that delay follows scene density in driving stacks. If the quadruped stack shows negligible coupling ($\beta \approx 0$), we will move the physical study to the driving setting and report the quadruped result as a negative finding.

The second step builds the coupled delay simulator. It extends MMDR's frame-buffer sampling so that the choice of which past frame to deliver is governed by the model above, driven by $c_t$ computed from simulator state.

The third step adapts PDTS so that its task identifier is $\xi$, jointly with physical parameters. It keeps PDTS's posterior sampling and diversity penalty, which matter here because hard delay tasks can collapse to "maximum $\beta$ everywhere." That collapse would reproduce worst-case padding.

The fourth step trains policies with PPO under each condition described in the Baselines section. For the high-dimensional delay-history input we optionally use a DT-CORL-style belief encoder, run online rather than offline.

Baselines

The naive reference is a policy trained with no delay modeling, the same no-delay baseline MMDR reports. That policy reached 444.7 cm average moving distance in the real Sparse Box test, per Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization.

The strongest existing method on the physical question is MMDR itself, which reached 859.9 cm in the same test. We run MMDR with uniform independent delays over its reported bounds and also over bounds widened to our measured P99. Both variants use the same PPO settings, environment steps, network, and hyperparameter search budget as our method.

The strongest tail-objective competitor is PDTS applied to independent delay parameters. The file reports only that PDTS improves $\mathrm{CVaR}_{0.9}$ by "over 15% on most Meta-RL benchmarks" relative to ERM and MPTS. No reported result exists on Lacuna for PDTS on any delay-randomized or latency task, so we must run it ourselves with the same sampler budget of pseudo-batches up to 64x.

We also include worst-case padding, in which every observation is delayed at the measured P99.

Two ablations isolate the key ingredient. The matched-marginal shuffle removes coupling while holding delay values fixed. Coupled delays with expected-return training and uniform sampling of $\xi$ removes the risk objective. Beating a baseline means a statistically separated improvement, over at least five seeds in simulation, in $\mathrm{CVaR}_{0.9}$ return and in collision rate over the worst 10% of episodes, while keeping mean return within a few percent. That last condition guards against buying safety with sluggishness, as COLA's fast path does.

Why not just…?

Why not just pad every observation with the worst-case delay? Padding makes the policy pay the tail cost in every state, including easy ones. The evidence points toward excessive caution. COLA's conservative fast path adds 6.43% commute time, and state-augmentation baselines in DT-CORL fall to near 0.2 on Adroit Hammer versus about 105 for belief modeling. We run padding directly and expect good tail safety with lower mean speed. If padding matches us on both counts, coupling adds nothing in that environment and we will say so.

Why not just use MMDR with wider, heavier-tailed independent delays? This is the real threat. With enough spread, independent sampling will sometimes place spikes in hard scenes. The matched-marginal shuffle is exactly this alternative, pushed to its best case, since it uses the identical delay values. If the shuffle performs as well as coupled training, our central claim fails. Only a gap concentrated in high-$c_t$ episodes supports it.

Why not just run PDTS over independent delay parameters? PDTS can find hard delay magnitudes, but it cannot express "delay grows with clutter" unless coupling parameters exist in its search space. Comparing PDTS-independent against PDTS-coupled, with the same sampler, separates the contribution of better task search from the contribution of a better delay model.

Why not fix latency in the systems layer instead? FogROS2-PLR concedes it gives only probabilistic P99 improvements at the cost of redundant carriers. Onboard compute contention of the kind COLA documents is not a network problem. As a stress test, we will deploy our policy on a stack whose worst latency spikes have already been reduced, to confirm the two approaches stack rather than conflict.

Proposed Experiments

We run simulation in three settings. The first is MMDR-style A1 obstacle locomotion with varied obstacle density. The second is CARLA driving with traffic-density scenarios, where COLA reports 1.65x worst-case latency gains and 93% of delay-caused accidents prevented at the systems level. We use these numbers as context only, since they come from a different intervention. The third is ManiSkill3 tasks from PDTS with clutter-driven perception delay, to check that the result generalizes beyond locomotion.

The primary metrics are $\mathrm{CVaR}_{0.9}$ return, collision rate among the worst 10% of episodes, and mean return. We also report performance binned by $c_t$ to show where gains occur. The physical test repeats MMDR's Sparse Box protocol and adds a dense-clutter variant, reporting moving distance and collision counts over repeated trials.

The decisive checks are these. The coupled policy must beat the matched-marginal shuffle specifically in high-complexity bins. It must match or exceed MMDR's mean moving distance while reducing worst-trial collisions. And removing the observation-age input must erase part of the gain, which would show the policy is using delay as a signal about the scene and not merely averaging over it.