Randomizing the Ground: Terramechanics Domain Randomization for Zero-Shot Off-Road Driving

Off-road robots fail in ways a camera cannot see coming. On packed dirt they grip, on loose sand they sink, and in deep snow or mud the wheels spin in place. Learning to handle these surfaces on real hardware means getting stuck, sliding off slopes, and breaking sensors, and there is little data because each hour of logs costs an hour of risky driving. Simulation is the usual escape, and domain randomization is the usual way to cross the reality gap. Domain randomization means training across many randomly perturbed versions of the simulator so the policy cannot overfit to any single one. Off-road sim-to-real work, however, randomizes only how the world looks. It does not randomize how the ground responds to a wheel.

We propose to randomize the wheel–soil interaction itself. A cheap semi-empirical force model would let sinkage, slip-dependent traction, and load- and speed-dependent rolling resistance vary across episodes. A teacher policy trained with access to the true soil parameters would then be distilled into a student that must infer them from a short history of IMU and wheel-speed signals, alongside a randomized depth-and-color terrain map. The claim is that this student transfers zero-shot, with no real training data, to deformable terrain where both vision-only sim-to-real policies and rigid-friction randomization fail. It should also avoid the sluggish, conservative behavior that very wide randomization usually produces.

Core Claim

The closest existing system is Sim-to-Real via Sim-to-Seg. It trains a reinforcement-learning policy in heavily visually randomized Unity scenes, translates images into canonical segmentation and depth, outputs waypoints tracked by a PID controller, and drives a full-size Polaris RZR zero-shot. That system uses rigid-body contact, gives the policy no proprioception, and trains the policy only on what a real sensor could see. Our proposal keeps its waypoint action abstraction, which motivates insulating the policy from low-level actuator errors. It changes three things. The simulated ground becomes deformable and randomized. The policy sees a proprioceptive history. Training runs asymmetrically, with privileged soil parameters available to the teacher.

Real-world self-supervised systems are the other neighbors. WayFAST estimates a traction coefficient online and projects it onto images. How Does It Feel? learns costs from IMU vibration power. Wild Visual Navigation adapts a traversability head within minutes of deployment. All three show that proprioception carries the signal that matters on deformable terrain. All three also have to experience the bad terrain on real hardware before they can learn it. We aim to move that experience into simulation.

The specific, falsifiable claim concerns the source of the benefit. It should come from the deformable-soil randomization, not merely from adding an IMU history or widening rigid friction ranges. The experiments are built to separate these three factors.

Research Program

Simulator. The simulator attaches a per-wheel terramechanics force law to a standard rigid-body vehicle model. The force law is of the pressure–sinkage and shear–displacement family, evaluated in closed form per wheel per step, so it adds negligible cost compared with discrete-element soil simulation. Discrete-element simulation is too expensive for reinforcement-learning sample counts, as the file notes. Randomized parameters include sinkage stiffness and exponent, cohesion, internal friction angle, and shear modulus. They also include patch-level spatial variation, so that one scene can mix firm and soft ground. Standard dynamics randomization of mass, center of mass, latency, and actuator limits follows Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. Visual randomization of textures, lighting, and weather follows Sim2Seg and (CAD)$^2$RL, the latter supporting diversity over photorealism.

Soil appearance is deliberately made only partially predictive of soil parameters. This mirrors reality, where identical-looking snow can be packed or deep, and it forces the student to rely on proprioception when vision is ambiguous.

Training. Training has two stages, following the privileged-teacher pattern from legged sim-to-real (Sim-to-Real: Learning Agile Locomotion For Quadruped Robots).

In the first stage, a teacher receives the true soil parameters $\xi$ under the wheels and in a look-ahead window. The teacher compresses them into a low-dimensional code $z^\ast = f(\xi)$ and learns a waypoint policy by reinforcement learning. Its reward trades progress to goal against getting stuck, rollover, and roughness.

In the second stage, a student encoder $E$ reads the last 0.5–1 s of IMU and wheel-speed signals $h_t$ plus the randomized terrain map. It is trained with

$$\mathcal{L} = \lVert E(h_t, m_t) - z^\ast_t \rVert^2 + \lambda\, \mathcal{L}_{\text{BC}},$$

where $m_t$ is the terrain map, $\mathcal{L}_{\text{BC}}$ is a behavior-cloning loss toward teacher waypoints on student-visited states, and $\lambda$ weights the two terms.

The low-dimensional IMU latent echoes Learning Inverse Kinodynamics, which learned a 2D terrain embedding from 30 minutes of real data. We instead learn it entirely from simulated soil variation.

Over-conservatism. Understanding Domain Randomization and How to pick the domain randomization parameters warn that wide randomization tends to produce timid policies. Our hypothesis is that inferring the soil code lets the policy be aggressive on firm ground and careful on soft ground, rather than careful everywhere. We will measure this directly as speed on firm terrain.

Coverage check against real logs. Before any field trial, we test whether real vehicle behavior falls inside the randomized distribution at all. TartanDrive and TartanDrive 2.0 provide synchronized IMU, wheel speed, shock travel, controls, and state on an ATV at up to 15 m/s. For each real window, we fit the simulator's soil parameters by minimizing one-step prediction error. We then report the fraction of windows whose residual falls below the simulator's own noise floor. Windows the simulator cannot explain show where the soil model is wrong.

Baselines

Naive reference. The do-nothing point is a kinematic vehicle model with a geometric elevation-map cost and MPPI tracking, with no learning and no soil model. For offline prediction it is the kinematic model TartanDrive reports at state-prediction RMSE of 1.1638. On deformable ground, the geometric LiDAR baseline in WayFAST reached 0% success in deep snow. We run it with the same waypoint tracker and sensors as our system.

Strongest existing zero-shot method. Sim2Seg is the strongest competitor on the exact question, because it is the only zero-real-data off-road driving system in the file. It reports an L2 distance to goal of 0.287 m on Arroyo Seco Trails, versus 0.653 m for a classical modular stack. We re-implement it in our simulator with the same visual randomization, the same waypoint action space and PID tracker, and the same reinforcement-learning sample budget and hyperparameter search budget. It trains on rigid ground, as in the original. Beating it means higher success and fewer stuck events on deformable test terrain, without losing on firm trails where it already works. If we lose on firm trails, the price of our added model must be reported.

Strongest real-data method. WayFAST reached 80% success in deep snow after training on real rollouts. We run a WayFAST-style traction-projection model trained on a matched amount of real driving on the test platform. It serves as a reference for what real data buys, not as a zero-shot competitor. Matching it with zero real data would count as success.

Ablations isolating the key ingredient. Four ablations each remove one factor. The rigid-DR ablation keeps everything, including the IMU history and teacher–student training, but replaces the terramechanics law with randomized Coulomb friction plus randomized linear rolling resistance and wheel damping. Its ranges are tuned on the same budget to cover the same range of stopping distances. The no-proprioception ablation removes $h_t$. The symmetric ablation trains the student directly by reinforcement learning without a teacher. The narrow-randomization ablation shrinks soil ranges to probe the conservatism trade-off. The rigid-DR ablation is the decisive one.

Why not just…?

Why not just widen rigid friction randomization? This is the most serious threat. If Coulomb friction and rolling-resistance randomization, combined with an IMU history, reproduce real stuck and slip behavior well enough, the terramechanics model adds nothing. The physical argument against it is that Coulomb friction cannot produce sinkage that grows with load and dwell time, or traction that depends on slip ratio. Those are exactly the effects that immobilize vehicles in snow and sand. Argument is not evidence, though. The rigid-DR ablation, run on an identical budget, settles the question both offline, through TartanDrive coverage and prediction RMSE, and in the field. If it matches our method, we will report that the benefit comes from proprioception and asymmetric training, and narrow the claim accordingly.

Why not just collect real data with self-supervision, as WayFAST or How Does It Feel? do? These methods work. The file notes, however, that TartanDrive 2.0 lost front shock sensors to aggressive driving, and that such methods learn a hazard only after driving into it. Label reprojection also degrades under the odometry drift that heavy slip causes. Our comparison measures how much real driving the WayFAST-style model needs to match a zero-shot student. It also counts stuck events incurred during that collection. If a few minutes of real driving suffices, the case for simulation is weaker, and we will say so.

Why not just deploy online adaptation, as Wild Visual Navigation does? Wild Visual Navigation adapts in 2–5 minutes on a quadruped that can recover from missteps. A wheeled vehicle that sinks into a drift often cannot recover. Online adaptation is, however, complementary rather than competing. We test it as an add-on initialized from our student and report whether the simulated prior shortens adaptation time or reduces early failures.

Why not just use a classical modular stack? Sim2Seg's own comparison, 0.287 m versus 0.653 m, and WayFAST's 0% LiDAR result in snow already indicate the modular stack's limits. It remains in our experiments as the naive baseline.

Proposed Experiments

Stage 1: offline test on TartanDrive, no hardware. We train a dynamics predictor purely in simulation. At test time, it infers the soil code from real IMU and wheel-speed history and predicts real ATV state transitions, following the TartanDrive evaluation protocol. The reference points are the kinematic model at 1.1638 RMSE and TartanDrive's real-data multimodal model at 0.1674 RMSE, which serves as an upper reference. We will re-run both under our exact splits so that the comparison is fair. The decisive check is whether the terramechanics-randomized predictor beats the rigid-DR predictor by a clear margin on the high-slip and soft-ground subsets. We will identify these subsets from wheel-speed-to-GPS discrepancy. The coverage fraction is reported alongside.

Stage 2: closed-loop simulation on held-out soil families. We test on held-out soil families, meaning parameter regions and spatial patterns excluded from training. Metrics are goal success, stuck rate, rollover rate, and average speed on firm segments. The firm-segment speed measures conservatism.

Stage 3: field trials. We evaluate on a mid-size wheeled robot across four terrain types: packed trail, dry sand, wet mud or grass, and snow if seasonally available. Each policy gets repeated trials per route. Metrics are success rate, human interventions per kilometer (as in How Does It Feel?), stuck events, and L2 distance to goal (as in Sim2Seg).

What would falsify the idea. Stage 1 alone can falsify the core claim before any hardware risk. If the rigid-DR ablation matches the full system in both Stage 1 and Stage 3, the claim is falsified. If the student becomes no faster on firm ground than the narrow-randomization policy, the conservatism hypothesis is falsified. If coverage on TartanDrive is poor, the soil model is inadequate and must be revised before field work. A final small-data ablation mirrors Sim2Seg's observation that about 400 real transitions helped. It asks whether a similar amount of real data, used only to shift the soil-parameter distribution, closes any remaining gap more efficiently than fine-tuning the policy.