Replaying What Transfers: Reality-Gap-Aware Replay for Turn-Heavy Robot Navigation

When a robot learns to navigate in simulation, the simulator is least trustworthy during turning maneuvers, the moments where real wheels slip, motors lag, and the robot drifts. A replay buffer is the memory of past experience that a learning agent re-trains on. Standard prioritized replay spends its attention on whatever surprised the agent most, measured by how wrong its value prediction was. In simulation, much of that surprise sits in sharp maneuvers that will not behave the same way on hardware. We propose to measure, from a small amount of real driving data, how badly the simulator mispredicts each kind of maneuver. That estimate is then used twice. The replay buffer discounts surprise that comes from untrustworthy physics. The policy pays a turning cost that depends on the state and is learned from the data: it is high where turns do not transfer and near zero where they do.

The motivating evidence is the "sim-to-real flip" documented in the first BARN navigation challenge (BARN Challenge at ICRA 2022). A deep reinforcement learning (RL) policy, DRL-VO, ranked first in Gazebo simulation with a score of 0.2415 but completed only 2 of 9 physical trials on a Clearpath Jackal. A classical hierarchical planner with latency compensation completed 8 of 9. According to that report, the failures involved actuator latency, footprint mismatch, and drift during tight maneuvers. The opportunity is an RL pipeline whose training data and training signal are both shaped by where the simulator can and cannot be believed. Such a pipeline would need only a small, deliberately collected set of real transitions.

Core Claim

The reality gap in constrained ground navigation is concentrated in heading changes rather than spread uniformly across states. If that holds, a cheap mismatch model trained on a small set of real transitions can do two things. First, it can separate surprise that is worth learning from, the informative temporal-difference error (TD error, the gap between predicted and bootstrapped value), from surprise caused by simulator artifacts. Second, it can supply a learned turn cost that replaces the fixed turn penalties of classical planners. Together these should transfer to hardware better than either reward-level turn penalties or grounding the simulator with ordinary replay.

Concretely, let $m(s,a) \ge 0$ be the predicted magnitude of the discrepancy between the simulator's next state and the real next state for state $s$ and action $a$. It is learned by regression on paired sim/real transitions. We set the replay priority of stored transition $i$ to

$$p_i = |\delta_i|^{\alpha}\, \exp(-\kappa\, m_i),$$

where $\delta_i$ is its TD error, $\alpha$ is the usual prioritization exponent, $m_i = m(s_i, a_i)$, and $\kappa \ge 0$ controls how strongly untrustworthy transitions are discounted. Importance-sampling corrections are kept as in standard prioritized replay. The policy is trained on a task reward minus $\lambda\, m(s,a)\, |\Delta\theta|$, where $|\Delta\theta|$ is the heading change the action produces. Turning is therefore charged only in proportion to how poorly the simulator models turning in that situation. A tight in-place rotation that the simulator models well costs little. A fast arcing swerve that slips on hardware costs a lot.

Relation to Prior Work

Three replay papers supply the mechanism and the tension. Prioritized Experience Replay samples transitions in proportion to TD error and reports beating uniform DQN on 41 of 49 Atari games at roughly double the learning speed. It also acknowledges that TD error is a poor proxy for learning potential under stochasticity, which is exactly the regime a simulator-reality mismatch creates. Reliability-Adjusted Prioritized Experience Replay (ReaPER) discounts TD error by how reliable the bootstrap target is, estimated from downstream TD errors along the trajectory. It reports reaching peak scores 16.6% to 32.6% faster than standard prioritized replay on classic control. Our discount is closest in form, but its source is different. ReaPER asks whether the value target is trustworthy yet. We ask whether the physics that produced this transition will exist on the robot, a question ReaPER cannot answer without real data. Experience Replay Optimization learns a replay policy and finds that favoring low-TD-error transitions often stabilizes continuous control. This supports the idea that high surprise is not always worth chasing. We impose a physically grounded reason for down-weighting rather than meta-learning one.

On the transfer side, Reinforced Grounded Action Transformation (RGAT) trains an action transformer by RL so that simulated transitions match a small set of real ones. It reports matching policies trained directly on the target environment in MuJoCo, using ordinary replay. It supplies both our grounding component and our main competitor. The question is whether replay and reward shaping that know where grounding is incomplete add anything once the simulator is already grounded. Domain randomization, as in (CAD)²RL and Control of a Quadrotor with Reinforcement Learning, constrains the design. It works for zero-shot transfer, but the file notes it yields conservative policies in tight spaces, and BARN courses have centimeter margins.

Turn minimization supplies the structural prior. Link* makes turns a primary cost in grid search and cuts turns to under 60% of Theta*'s at up to 122% of its path length. Turn-penalized A* for tracked machinery (Path Planning and Tracking Control of Tracked Agricultural Machinery Based on Improved A* and Fuzzy Control) reports 39.01% fewer turning points on field vehicles. Both use a fixed, hand-set turn cost on known maps. We keep their insight that turns deserve their own cost and replace the constant with one estimated from the robot's own mismatch data. The later BARN reports show what the bar is. KUL+FM's swept-volume planner achieved 9/9 (2nd BARN Challenge), and all teams needed on-site tuning. LiCS-KI won 6/9 with a behavior-cloned Transformer and a geometric safety layer (3rd BARN Challenge), bypassing online RL entirely.

Research Program

The first stage is diagnostic, and the rest of the program depends on it. We build a sim-to-sim proxy of the reality gap in BARN's Gazebo worlds. The "real" robot is a Jackal model with injected command latency, slip that grows with lateral acceleration, motor deadband, and a slightly enlarged footprint. For MuJoCo, we reuse RGAT's modified-dynamics protocol on Hopper and HalfCheetah. We then measure how the simulator's one-step prediction error is distributed across maneuver types (straight, in-place rotation, arcing turn, as functions of speed and heading rate). We repeat the measurement on a few hours of teleoperated real Jackal driving. If the error is not concentrated at heading changes on real hardware, the turn-cost half of the proposal is dropped. What remains is a general mismatch-aware replay method, and we will report it as such.

The second stage builds the method. The mismatch model is a small ensemble regressor trained on paired transitions, meaning a real transition and the simulator's replay of the same state and action. Physical data collection is scaffolded by a Link*-style low-turn planner: straight segments are cheap to collect safely, and turn samples are requested specifically where the ensemble disagrees. Real-robot time goes to the maneuvers the simulator models worst. The policy is an off-policy actor-critic trained in the RGAT-grounded simulator with the priority and reward defined above. The mismatch model and the grounding transformer are periodically refit as new real data arrives. The third stage is hardware evaluation on BARN-style cardboard courses.

Baselines

The naive reference is the ROS Dynamic Window Approach, which the 2022 report says all finalists beat. The pure-RL reference is an off-policy agent trained with uniform replay in the ungrounded simulator. It plays the role DRL-VO played, and DRL-VO's reported physical result is 2 of 9 trials (BARN 2022).

The strongest competitor on the scientific question is RGAT grounding plus standard TD-error prioritized replay, and separately RGAT plus ReaPER. It uses exactly the same real-data budget, the same grounded simulator, the same actor-critic, network size, and tuning budget, and it isolates whether our mismatch-aware weighting adds anything beyond grounding. The file reports no numerical RGAT result, only near-optimal transfer matching a target-trained policy ("no reported result found on Lacuna" for a specific RGAT score). In MuJoCo, beating it means reaching the target-environment return with fewer real transitions, or reaching a higher return at a fixed real budget. The comparison uses at least ten seeds with confidence intervals.

The strongest existing hardware results are KUL+FM's 9/9 classical planner (BARN 2023) and, among learned methods, LiCS-KI's 6/9 with traversal times as low as 27 s (BARN 2024). Our hardware target is to exceed 6/9 as a learned method without on-site manual tuning, and to beat the grounded-RGAT baseline under an identical real-data budget. We do not claim to beat 9/9. We will rerun a behavior-cloning baseline of the LiCS-KI type on our own courses, because challenge scores come from different sites.

Two ablations isolate the ingredients. One sets $\lambda = 0$, keeping replay discounting only. The other sets $\kappa = 0$, keeping only the learned turn cost. A third replaces $m(s,a)$ with a constant, recovering a fixed Link*-style turn penalty. This checks whether a state-dependent estimate matters at all.

Why not just…?

Why not just add an angular-velocity penalty to the reward? This is the most serious threat, because if the reality gap is uniform across turns, a constant penalty captures most of the benefit at no cost. The file's argument against it is that BARN requires sharp in-place rotations in narrow gaps, so a blanket penalty discourages exactly the maneuvers that succeed. That argument is plausible but untested. The constant-$m$ ablation settles it directly. If a tuned constant turn penalty matches our success rate and real-data efficiency, the learned turn cost is unnecessary and we will say so. The replay component would then stand or fall on its own ablation.

Why not just randomize friction, latency, and mass broadly? Randomization needs no real data, which is a real advantage. The file notes it produces sluggish, conservative policies in tight spaces, and DRL-VO's 2/9 shows that simulation training alone did not suffice on BARN. We run a domain-randomization baseline over the same latency and slip ranges we inject in the proxy. We expect lower traversal speed and success in the densest worlds, and we report the outcome whichever way it falls.

Why not just use RGAT with ReaPER, since ReaPER already discounts unreliable TD errors? ReaPER's reliability comes from downstream TD errors within the agent's own training. It is blind to whether a confident, well-bootstrapped value was learned on physics that the robot lacks. Once grounding is applied, though, the residual mismatch may be too small to matter. This is exactly the RGAT+ReaPER baseline, and the $\lambda = 0$ ablation measures the margin.

Why not just behavior-clone a classical planner, as LiCS-KI did? Cloning avoids exploration cost. However, it cannot exceed its demonstrator, and LiCS-KI still failed 3 of 9 trials in tight corners. We include it as a baseline. If it matches us on success and speed with less effort, the case for online RL on BARN is weak, and we would report that.

Decisive Checks

Three results decide the proposal. First, the diagnostic must show that real one-step prediction error is substantially concentrated at heading changes. Without this, the turn framing is abandoned. Second, at a fixed real-transition budget, the full method must beat RGAT plus prioritized replay and RGAT plus ReaPER in the latency-and-slip proxy. It must also beat the constant-turn-cost ablation, which shows that a learned turn cost beats a Link*-style fixed one. Third, on the hardware courses, the method must exceed the 6/9 learned-method mark under the 9-trial protocol without on-site tuning. Alongside success, we report traversal time, turns per run, and the fraction of replay priority mass assigned to high-mismatch transitions. The last quantity shows whether the mechanism is doing what we claim.