Randomizing the Sensor Where the Policy Sees It: State-Space Domain Randomization for Dense-Sensing Robots

Robots trained in simulation often fail on real hardware because real sensors differ from simulated ones. Real depth cameras have holes, edge artifacts and correlated noise, and real LiDAR returns are messier than rendered ones. The usual fix, domain randomization, means making the simulator's sensors deliberately noisy and varied so the learned policy cannot rely on any particular quirk. For dense sensors this is expensive, because thousands of pixels must be rendered and corrupted every step, and it is guesswork, because nobody knows which pixel-level corruptions matter. This proposal starts from a fact in fast state estimation. When a filter fuses a dense measurement by projecting its residuals into "Jacobian space" (multiplying the per-pixel errors by the measurement Jacobian, the matrix that says how each pixel would change if the robot or object moved), the thousands of numbers collapse into a small summary whose size is set by the state, roughly fifteen numbers for a pose-and-velocity state. Everything the downstream controller ever learns from the sensor passes through that summary.

The idea is to do domain randomization in that small summary instead of in raw pixels. If the gap between simulated and real sensing is fully expressed in the state-sized summary, then a few physically meaningful perturbations of it can reproduce the real distribution of filter errors and filter confidence. Those perturbations are an overconfidence factor, a systematic bias, and the loss of observability along some directions. They can be fitted from a small amount of real logged data with ground truth, then randomized cheaply across millions of simulated episodes. A policy that reads the filter's estimate and its covariance (the filter's own error bars) can then learn to trust those error bars, because during training they were miscalibrated in the same ways they will be on the robot.

Core Claim

Dense-sensor filters that update in state space reduce each frame to an information pair. With $H_i$ the Jacobian of pixel or point $i$, $r_i$ its residual and $R_i$ its assumed noise variance, the pair is

$$\Lambda = \sum_i H_i^\top R_i^{-1} H_i, \qquad \eta = \sum_i H_i^\top R_i^{-1} r_i,$$

and the update is $P^+ = (P^{-1} + \Lambda)^{-1}$, $\delta x = P^+ \eta$, where $P$ is the prior state covariance and $\delta x$ the state correction. This is the computation behind the factorized per-pixel update of Depth-Based Object Tracking Using a Robust Gaussian Filter, the state-dimensioned gain of FAST-LIO2, and the Jacobian-space projection of SchurVINS. We claim that the sim-to-real sensing gap, seen through this pair, is well described by a low-dimensional perturbation family:

$$\tilde\Lambda = \Lambda/\kappa, \qquad \tilde\eta = \eta + \Lambda b + \epsilon,\ \ \epsilon \sim \mathcal{N}(0, (\kappa - 1)\Lambda),$$

possibly followed by deleting the information along a sampled direction. Here $\kappa \ge 1$ models overconfidence. Neighbouring pixels are not independent, so the true information is smaller than the sum assumes. The vector $b$ is a state-space bias, for example a depth offset or a miscalibrated extrinsic. The noise term $\epsilon$ is constructed so that the perturbed residual is consistent with the deflated information. Sampling $(\kappa, b, \text{degeneracy})$ per episode from a distribution fitted to real logs is our state-space domain randomization. It differs from injecting pose errors because it perturbs error and reported covariance together and in the right coupled way. It differs from pixel randomization because it costs $O(N^2)$ per step and is calibrated against real data rather than hand-designed.

Relation to Prior Work

Three estimation papers supply the mechanism. Issac et al. sum independent per-pixel contributions into state-space statistics and track objects at 30 Hz from 3072-dimensional depth observations. FAST-LIO2 refactors the gain so that cost scales with state dimension, which lets it register raw LiDAR points at 100 Hz on an i7 and 10 Hz on an ARM board. SchurVINS replaces QR decomposition with direct Jacobian-space projection, reaching 0.075 m average error on EuRoC at 3.83 ms per frame on MH01. All three are evaluated only as estimators on static benchmarks. None asks what a controller downstream needs from the projected quantities, or how those quantities shift between simulation and reality. We reuse their computation unchanged and study its interface to learning.

Domain randomization motivates the problem. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization and Sim-to-Real: Learning Agile Locomotion For Quadruped Robots randomize physics and add unstructured Gaussian noise to low-dimensional observations. (CAD)$^2$RL randomizes rendered appearance. AdaptManip uses a learned recurrent network (an LSTM) as a black-box object-state estimator under dynamics randomization. These works randomize either the physics or the raw sensor. We add a third place to randomize: the estimator's state-space output. We keep standard dynamics randomization alongside it.

The closest existing work is FalconGym. It fits perception-error distributions with quantile regression and injects them into controller training, with a Kalman filter fusing a neural pose estimator with an IMU, and reports 95.8% zero-shot gate success. That work constrains our design. It shows that error injection downstream of perception can work, so our contribution must come from coupling errors with covariances and from fitting a structured family, not from injecting errors per se. Its errors are perturbations of a pose output. The filter there consumes poses rather than dense residuals, and the controller never sees a covariance that is right or wrong in a calibrated way.

Asymmetric Actor Critic supplies the training scaffold. In that method the critic, the network that scores actions during training, sees privileged simulator state while the actor, the network that picks actions, sees only deployable inputs. We keep that structure, but the actor's input becomes the filter mean and covariance rather than raw images. A Differentiable Extended Kalman Filter for Object Tracking Under Sliding Regime shows that learned components can live inside a filter, but it trains on real data with a few tactile channels. We need no gradients through the filter and target dense sensors.

Research Program

The first stage is measurement. On real logged sequences with ground truth, we run the projected-residual filter and record $(\Lambda, \eta)$ alongside the true state error. We also render the same trajectories in simulation, with simple ideal-noise depth or LiDAR, and record the simulated pair. We then fit the perturbation family by maximizing the likelihood of real filter errors given simulated information pairs, and test how much of the real error distribution it explains. The key diagnostic is filter consistency: the normalized estimation error squared (NEES), which is the state error weighted by the inverse reported covariance. NEES should equal the state dimension if the reported covariance is honest. If a handful of parameters bring simulated NEES statistics and error spectra in line with real ones, the core claim survives. If real errors require pixel-level structure that cannot be summarized in state space, it does not. Examples would be failures that depend on scene layout in ways $\Lambda$ cannot see, such as wrong data association.

The second stage is policy training. We take a tabletop manipulation task modeled on the Fetch puck-pushing setup of Peng et al., but the puck and object state come only from a wrist or head depth camera passed through the Issac-style factorized filter. We train in MuJoCo with dynamics randomization as in that paper. The actor receives the filter mean and a compact encoding of $P$, and the critic receives true state. During training we compute the cheap ideal-noise information pair for each step, perturb it with sampled $(\kappa, b, \text{degeneracy})$, and run the filter update on the perturbed pair. The rendering stays cheap because pixel corruption is never simulated. We also randomize the fitted distribution's width, to address the conservatism-versus-coverage tension noted for domain randomization.

The third stage checks transfer on a second embodiment. A small aerial or mobile platform runs FAST-LIO2-style LiDAR-inertial odometry for a waypoint or gate-tracking task. There the dominant real failure is degeneracy in feature-poor corridors, which our family models as information deletion along a direction.

Baselines

The naive reference is a policy trained with dynamics randomization only, which reads the unperturbed filter mean from ideal simulated sensing and ignores covariance. It measures how much the sensing gap costs when nothing is done about it.

The strongest existing competitor for the low-dimensional control part is the recurrent dynamics-randomized policy of Peng et al. That paper reports a real success rate of 0.89 ± 0.06 on Fetch puck pushing from low-dimensional puck coordinates, against 0.67 for a feedforward randomized policy. We will reproduce its LSTM policy with the same randomization ranges and training budget, feeding it the same filter outputs our policy sees. Beating it means matching or exceeding its real success rate when puck state comes from the depth filter rather than clean coordinates, with a gap that widens under induced sensor degradation. For estimation quality we compare against the 4.89 mm median translational error reported by Issac et al. on their depth tracking benchmark. We will reproduce that filter on the same sequences, since our filter is that filter with recalibration, and our perturbation model should not worsen it.

The second strong competitor is FalconGym-style error injection. We fit quantile-regression pose-error distributions from the same real logs and inject them into the filter mean during training, with the filter's nominal covariance passed through unchanged. This uses identical data and an identical training budget. It is the crucial ablation, because it keeps the real-data fitting and drops only the coupling between error and covariance.

A third baseline is pixel-level randomization. Here we render depth with hole patterns, edge noise and correlated noise fields, and run the full filter in simulation. It has the same number of environment steps, and its added wall-clock cost is reported. A final ablation removes the covariance from the actor's input, to isolate whether the policy actually uses calibrated confidence.

Why not just…?

Why not just train an LSTM on depth images with ordinary domain randomization, as AdaptManip and Peng et al. do? Recurrent policies infer hidden variables well when observations are low-dimensional. Peng et al.'s 0.89 is on joint angles and puck coordinates. Pushing thousands of depth pixels through a recurrent network at control rate on embedded hardware is a different regime, and the network's internal state gives no calibrated uncertainty. We include the LSTM on filter outputs as a baseline. We also include an end-to-end depth-image recurrent policy in simulation, reporting its sample count and inference latency. If it matches our real success at comparable cost, the structured interface is not justified.

Why not just inject pose errors fitted to real data, as FalconGym does? This is the real threat, and it is the main ablation. Our hypothesis is that error injection without covariance coupling teaches the policy that the reported covariance is meaningless, so it cannot behave differently when the estimate is degenerate. The test is a degeneracy-heavy condition, such as an occluded object face or a featureless corridor. If pose-error injection matches our method there as well, we will conclude that coupling adds nothing and report the method as a cheaper equivalent at best.

Why not just render realistic sensor noise in simulation? Pixel randomization is the honest approach, and it may work. Its weaknesses are cost, since every parallel environment renders and corrupts dense frames and runs the full update, and calibration, since it is unclear which corruptions matter. The decisive comparison is real success per unit of simulation compute, together with the stage-one consistency test: whether pixel-randomized simulation or our fitted state-space family better reproduces real NEES distributions.

Why not just downsample until an ordinary filter suffices? FAST-LIO2 reports that aggressive downsampling or feature extraction fails in feature-poor scenes and under aggressive motion. Downsampling also does not touch the sim-to-real question, which is about miscalibrated confidence rather than speed.

Proposed Experiments

The decisive offline check uses the 37-sequence depth tracking benchmark of Issac et al. and the EuRoC MAV dataset. We fit $(\kappa, b, \text{degeneracy})$ on half the sequences and test on the rest, reporting held-out NEES calibration, error-quantile match and log-likelihood against pose-error quantile models. On EuRoC no matched simulated rendering exists, so we take the ideal-noise baseline as the filter's nominal Jacobian-space information with an independent-pixel noise assumption. We then test whether deflating and biasing it explains the real errors. Failure here, for example a $\kappa$ that varies wildly across scenes with no predictable pattern, ends the project before any robot time.

The decisive online check is zero-shot real pushing and pick-and-place with depth-only object state. We run twenty or more trials per method, in nominal conditions and under induced degradation: partial occlusion, reflective tape on the object, and a perturbed camera extrinsic. Metrics are success rate, final object position error, and how often the policy takes information-gathering actions, such as re-orienting the camera, when covariance is large. The LiDAR platform repeats the comparison on corridor and open-area courses, measuring tracking error and failure rate.

The idea is supported if three things hold together: our method beats pose-error injection specifically in the degraded conditions, matches pixel randomization at a fraction of its compute, and removing covariance from the actor erases the gain.