Searching Robot Policy Architectures Under Their Own Inference Delay

A neural network that controls a robot is a computer program running on a small onboard chip, and running it takes time. A deeper or wider network may produce better actions, but it produces them later, so the robot acts on a slightly stale view of the world. Today, simulators used to train these controllers either ignore this delay or insert one fixed delay measured once on the robot. Engineers then hand-pick a tiny network that they trust to be "fast enough." We propose to make the delay a consequence of the network's design. For every candidate architecture, a fast hardware cost model predicts how long that network will take to run on the target chip. The parallel GPU simulator then delays that candidate's actions by exactly that amount while it trains. The search keeps the architecture that performs best with its own delay applied.

The claim is that this changes which architecture wins. In feedback control, delay does not cost a fixed amount the way an energy bill does. It erodes stability, and how much it hurts depends on how agile the task is. Searching with an "accuracy minus λ·latency" score, the standard practice in hardware-aware architecture search, should therefore pick different and worse controllers than searching with the delay inside the simulated physics. GPU simulators now train a locomotion policy in seconds to hours, which makes it affordable to train many candidates. That cost, not the idea, is what used to rule this out.

Core Claim

Let $\alpha$ be a policy architecture and $h$ a target compute platform. Let $\hat{\ell}(\alpha, h)$ be the predicted inference latency, as a distribution rather than a point. Let $R(\pi \mid \ell)$ be the episodic return of policy $\pi$ when each action takes effect $\ell$ after the observation it was computed from. Standard hardware-aware search scores a candidate as $R(\pi_\alpha \mid 0) - \lambda\,\mathbb{E}[\hat{\ell}]$. We score it as

$$F(\alpha) = R\big(\pi_\alpha^{\ast} \mid \ell \sim \hat{\ell}(\alpha, h)\big),$$

where $\pi_\alpha^{\ast}$ is the policy trained in a simulator whose actions are delayed by draws from $\hat{\ell}(\alpha, h)$. The two scores agree only if delay affects return additively, and control theory says it does not: delay shrinks the margin before oscillation in a nonlinear, task-dependent way. Our testable predictions follow from this. The two scores should rank architectures differently. The difference should grow with task agility, for example sprinting versus walking. Architectures chosen by $F$ should transfer better to a held-out "reality" with realistic, jittery latency.

Positioning Against Prior Work

The motivating evidence comes from sim-to-real robotics. Sim-to-Real: Learning Agile Locomotion For Quadruped Robots showed that zero-shot galloping on the Minitaur required explicitly modeling a 15–19 ms onboard control-loop latency alongside motor dynamics. The policy reached 1.18 m/s in the real world against 1.34 m/s in simulation. That paper treats the latency as a measured constant, and its network is a fixed hand-designed MLP. We make latency a function of the architecture and search over the architecture. Humanoid-Gym runs a hand-set 100 Hz policy over a 1,000 Hz PD controller and validates its policies in MuJoCo before deployment. Rapid Locomotion via Reinforcement Learning hand-designs teacher and student networks that reach 3.9 m/s on the Mini Cheetah. Both treat compute timing as outside the learning problem. We reuse Humanoid-Gym's sim-to-sim validation idea as our proxy for reality.

The method comes from two other lines. GPU-native simulators keep physics and policy in the same device memory and step thousands of environments as one tensor operation: Isaac Gym reports up to 700K steps/s, and VMAS applies the same approach to multi-agent scenarios. Fast Population-Based Reinforcement Learning on a Single Machine trains populations of agents together on one machine. We need the throughput from the first two and the population training from the third. Analytical cost models estimate the run time of a network from its layer shapes and the hardware's dataflow without running cycle-level simulation. MAESTRO does this within 3.9% of RTL simulation at about 10 ms per layer. Layer-wise lookup tables like those in A Multi-objective Optimization Benchmark Test Suite for Real-time Semantic Segmentation and FPGA/DNN Co-Design do the same by summing pre-measured per-layer costs. We reuse these models as the latency source.

The closest competitors are hardware-aware search methods: APQ, Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Accelerator, and benchmarks such as HW-NAS-Bench. They jointly choose a network and a hardware or compression configuration using predicted latency. All of them target open-loop image tasks, where latency can only enter the score as a penalty or a constraint. Our proposal differs in where latency enters: it is a physical delay in the simulated dynamics during training, not a term added to the score afterward.

The constraints come from known limits. MAESTRO itself notes that analytical models miss dynamic effects, and the file notes that they abstract away operating-system jitter, bus contention, and cache misses on embedded processors. We therefore predict a latency distribution calibrated from a few board measurements, not a point estimate. We also test directly whether analytical predictions are good enough. The evolutionary search results of Large-Scale Evolution of Image Classifiers suggest that simple mutation-based search is enough for us, so the search algorithm is not where the contribution lies.

Research Program

The first component is a delay-aware vectorized environment wrapper for Isaac Gym. Each parallel environment keeps a short ring buffer of past actions and a per-environment delay index. At every physics step, an environment applies the action computed $\ell$ milliseconds earlier, with $\ell$ drawn from its assigned architecture's predicted latency distribution. Because this is pure tensor indexing, environments with different delays share one batch with no divergent control flow.

The wrapper also models the second effect of slow networks: when inference latency exceeds the control period, the network must run at a lower rate. Each candidate therefore receives both a delay and an effective control rate, set by whether the target chip can pipeline inference. Many candidate architectures are then trained at once, each owning a slice of the environment batch, in the style of population-based training.

The second component is the latency predictor. For an embedded CPU target, it is a per-operator cost model of compute and memory traffic, fitted to on the order of tens of measured networks. For a small accelerator target, where hardware parameters such as the number of processing units and buffer sizes are themselves design variables and no physical board exists, we use a MAESTRO-style dataflow model. In both cases the output is a latency distribution: a point prediction plus an empirically fitted jitter term.

The third component is a deliberately simple evolutionary search over policy architectures. The space covers MLP depth and width, the length of the observation history, temporal-convolution versus recurrent history encoders, and, for the accelerator target, the hardware configuration. Every candidate is scored by $F(\alpha)$ after a fixed training budget.

The analysis then asks four things. Where do the delay-coupled and penalty-based rankings disagree? Is the disagreement explained by task agility? Does the selected architecture change as the control frequency or speed command changes? How much ranking error does the analytical latency model introduce relative to measured latency?

Baselines

The naive reference point is current practice. A hand-designed MLP of the size used in Rapid Locomotion via Reinforcement Learning is trained in Isaac Gym with one fixed delay measured on the target board, as in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. It receives the same total number of simulation steps as the entire search. Beating it means higher return in the proxy-reality evaluation described below at matched or lower measured latency.

The strongest existing method is hardware-aware search with an additive latency penalty, the scoring used by APQ and Best of Both Worlds, transplanted to reinforcement learning. Candidates are trained with zero delay and scored by return minus $\lambda$ times predicted latency. It runs with the identical search space, search algorithm, latency predictor, and simulation budget, and $\lambda$ is tuned over a grid using the same tuning budget we give our method. Its selected architectures are then retrained with their true predicted delay before evaluation, so it is not handicapped at deployment time. This is the strongest competitor because it already combines fast cost models with automated search. The only thing it lacks is our key ingredient, latency treated as a physical delay. There is no reported result found on Lacuna for hardware-aware architecture search on robot locomotion, so its numbers will be produced here. Beating it means a better proxy-reality return at every point of the return-versus-measured-latency Pareto front, across seeds.

A third baseline fixes the architecture search but trains every candidate with delay randomized over a wide range, for example 5–40 ms, instead of its own predicted delay. A fourth applies post-hoc compression: we train the largest network in the space and quantize and prune it to meet the latency budget, following the train-then-compress pipeline that APQ argues against for images.

Two ablations isolate the ingredients. The first replaces the analytical latency predictor with latencies measured on the real board for every candidate. The second removes jitter, keeping only the point prediction.

For reference, the throughput figures are Isaac Gym's report of Ant reaching performant locomotion in 20 seconds and Humanoid reaching reward 5,000 in under 4 minutes. These show per-candidate training is cheap enough for populations of hundreds.

Why not just…?

Why not just randomize action delay during training? This is the most serious threat, and the file's evidence against it is indirect. Wide randomization is known to make policies conservative (OpenAI et al. relies on it at the cost of very large training). The agile results in Rapid Locomotion and the Minitaur work relied on precise modeling rather than broad ranges. But randomization only changes how a policy is trained. It does not choose an architecture. The fair test is therefore our third baseline, which uses the same search with randomized delay. If it matches our method on the agile sprint tasks, the claim that architecture-specific delay matters is wrong, and we will report that. We expect randomization to tie on slow walking and lose on fast running, where conservatism is most costly.

Why not just penalize latency in the score? If delay degraded return roughly linearly, a tuned $\lambda$ would recover our rankings, and our method would add nothing. This is exactly what the penalty baseline tests. The decisive statistic is the rank correlation between the two scores across the population, broken down by task speed, together with whether any single $\lambda$ reproduces our selections across tasks. A high correlation would falsify the core claim.

Why not just measure latency on the real board instead of modeling it? For a fixed off-the-shelf board and a small MLP search space this is a genuine competitor. Measuring takes seconds, while training a candidate takes minutes, so the analytical model's speed advantage, which matters in MAESTRO's setting of 170,000 designs per second, is not decisive here. We therefore do not claim speed as the benefit for that case, and measured latency is an ablation, not a straw man. The analytical model earns its place in two places. The first is when the hardware itself is being designed, as in the accelerator experiment, where nothing exists to measure. The second is when predicting jitter and rate effects for boards not yet in hand. The ablation shows whether analytical predictions select the same architectures as measurement. If they do not, the predictor must be fixed before the co-design results mean anything.

Why not just train big and compress afterwards? Compression changes the network's outputs after training, and in a closed loop small output errors can accumulate into drift. The policy was also never trained under the delay the compressed network will actually have. The fourth baseline measures both effects directly.

Proposed Experiments

The tasks are Isaac Gym's Ant, Humanoid, and ANYmal locomotion suite, plus a Mini Cheetah velocity-tracking setup in the style of Rapid Locomotion. We sweep commanded speeds so that agility varies along a known axis.

The proxy reality follows Humanoid-Gym's sim-to-sim protocol. Policies are evaluated in MuJoCo with action delays replayed from inference-latency traces recorded by running each final network on the actual embedded board, so the evaluation carries real jitter that no training condition saw. Metrics are return, velocity-tracking error, fall rate, and the coefficient of variation and cost of transport used by A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learning. All results are reported over five or more seeds.

Three checks decide the proposal. First, the rank correlation between delay-coupled and penalty-based scores must fall noticeably as commanded speed rises. Second, our selected architectures must dominate the tuned-penalty and randomized-delay baselines on the proxy-reality Pareto front at high speeds. Third, searches driven by analytical and measured latency must select overlapping architectures.

A secondary experiment uses the MAESTRO-style accelerator model to co-search policy and processing-unit count under a power budget. It asks whether the cheapest hardware that keeps delay below the task's stability threshold differs from the hardware an additive score would pick.

If the transfer budget allows, the final step deploys the top architectures on a physical quadruped. That step would confirm the result but is not required for the core claim.