Filter Innovations as a Label-Free Test of Synthetic Training Data

Robots increasingly learn their state estimators in simulation. A differentiable filter is a classical tracking algorithm, such as a Kalman or particle filter, written so that a neural network inside it can be trained by gradient descent. These filters are typically trained on simulated trajectories and then deployed on real sensors. The weak point is evaluation. Deciding whether one simulator configuration produces better training data than another normally requires labeled real trajectories, and those are exactly what is expensive to collect. This note proposes using a signal the filter already computes at every step. That signal is the innovation: the gap between the measurement the filter predicted and the one it actually received, together with the uncertainty the filter attached to that prediction. If the synthetic data taught the filter a wrong noise model or a brittle sensor encoder, its innovations on real, unlabeled sensor streams should be statistically inconsistent. The surprises will be too large, too small, or correlated over time.

The claim is narrow and testable. A score built from innovation consistency on unlabeled real streams should rank synthetic training sets, which differ in generator settings, noise models, motion diversity, and amount, in nearly the same order as their true real-world tracking error and calibration. If it does, practitioners can evaluate and select synthetic data without real labels. Where it fails, we want to know which kinds of simulator errors are invisible to it.

Core Claim

Let $\nu_t = z_t - \hat z_t$ be the innovation at time $t$. Here $z_t$ is the measurement, or the learned encoder's pseudo-measurement, and $\hat z_t$ is the filter's predicted measurement. Let $S_t$ be the filter's predicted innovation covariance and $m$ the measurement dimension. The normalized innovation squared is

$$\epsilon_t = \nu_t^\top S_t^{-1} \nu_t .$$

For a statistically consistent filter, $\epsilon_t$ has mean $m$ and is uncorrelated across time. This follows from the standard Gaussian filtering result that consistent innovations form a white sequence with covariance $S_t$. We define a data-quality score for a filter trained on synthetic set $D$ and run on an unlabeled real stream:

$$C(D) = \left|\tfrac{1}{T m}\textstyle\sum_t \epsilon_t - 1\right| + \lambda \sum_{k=1}^{K} |\rho_k| ,$$

where $\rho_k$ is the lag-$k$ autocorrelation of the normalized innovations, $K$ a small number of lags, and $\lambda$ a weight fixed in advance. The proposal is that $C(D)$ predicts the real labeled error and negative log-likelihood of the filter trained on $D$. This makes it usable as a label-free metric for synthetic data, and as an estimator of how many synthetic trajectories one real trajectory is worth.

The theory also says where the score should go blind. Innovation consistency can only see errors that show up in the measurement. A constant bias in a learned visual encoder can be absorbed into the state estimate, leaving the innovations white while the trajectory is wrong. A central goal is to map this blind spot rather than hide it.

Positioning

Two papers in the file motivate this work. How to Train Your Differentiable Filter shows that training with negative log-likelihood gives calibrated, input-dependent observation noise. In its disc-tracking task, the learned noise grows from 0.9 to 29.3 under occlusion. That calibration was only ever checked in-distribution. The file's gap analysis notes that no one has measured covariance consistency of differentiable filters under sim-to-real shift. The second is On the Equivalency, Substitutability, and Flexibility of Synthetic Data, which measures a roughly 30:1 synthetic-to-real "exchange rate" for multi-person tracking. It needs labeled real test data to do so, and it studies a feedforward tracker rather than a recursive one.

The closest existing bridges train filters on synthetic data but never evaluate the data itself. Enhancing State Estimation in Robots pretrains an ensemble Kalman filter on 2,000 MuJoCo demonstrations and fine-tunes on 100 real ones. Particle Filter Networks trains and tests entirely inside House3D. Learning Interpretable BEV Based VIO without Deep Neural Networks trains its four noise parameters on CARLA or KITTI separately and never crosses domains. Deep Measurement Updates for Bayes Filters transfers a synthetic-only depth likelihood zero-shot to real articulated objects, but reports no tracking-error number in the file and does not judge data quality.

What Went Wrong? (COMPASS) is the nearest in spirit. It tunes simulator parameters by backpropagating trajectory discrepancies, raising air-hockey success from 0% to 80%. However, it works offline on physical parameters for policy learning. It does not use an online estimator's internal statistics, and it needs paired real trajectories. Our work differs from all of these by turning the deployed filter itself into the evaluator of its training data, using only unlabeled real sensor streams.

Research Program

The program has three stages.

In the first stage, we build a pool of synthetic training sets for each testbed. These are produced by controlled changes to the generator. The axes are the sensor noise model (Gaussian white versus heavy-tailed, biased, or drifting), motion diversity (planar versus 6-DoF; TartanAir's motion-diversity measure is 0.95 against KITTI's 0.005 per TartanAir), rendering condition (weather and lighting), and dataset size. For each set we train the same differentiable filter with negative log-likelihood, following the recipe of How to Train Your Differentiable Filter, including its roughly ten-step training sequences.

In the second stage, each trained filter is run on unlabeled real streams to compute $C(D)$. Separately, it is run on held-out labeled real streams to compute translation error, rotation error, and state negative log-likelihood. The main analysis is the rank correlation between the two across the pool.

In the third stage, we characterize failure modes. We inject known simulator errors into a controlled "real" domain: constant encoder bias, unmodeled process drift, and miscalibrated noise scale. Theory predicts the score should detect noise-scale and drift errors but miss biases the filter can absorb. We then test whether adding a second, independent sensor channel, such as IMU next to vision, restores sensitivity to bias. With two channels, a bias in one shows up as a disagreement the filter cannot absorb.

Baselines

The naive reference point is to choose synthetic data by default, using the largest or most photorealistic set. Equivalently, it ranks sets randomly. Any useful score must beat this clearly.

The strongest competitor, and the upper bound we compare against, is the labeled-real-validation oracle. It ranks each synthetic set by the trained filter's actual error on a small labeled real split. This is what a practitioner with labels would do. We give it a deliberately small labeled budget: the same number of real sequences that our method uses unlabeled, plus a few-sequence variant. This lets us ask whether unlabeled innovations match a modest labeled set.

For absolute accuracy context on KITTI, the strongest reported differentiable filter in the file is the differentiable particle filter dPF-M-lrn, at 14.3 m translation error versus 25.7 m for an LSTM (How to Train Your Differentiable Filter). On the UR5 sim-to-real task, DEnKF reports a 2.6 cm average end-effector offset (Enhancing State Estimation in Robots). For CARLA we found no reported numerical result on Lacuna. We do not aim to beat these accuracies. They are used to check that our reimplementations are within a reasonable range before ranking is studied.

The second competitor is a label-free distribution-distance score. It measures the distance between encoder feature statistics on synthetic versus real inputs, which is the usual unsupervised domain-gap measure. It uses the same unlabeled real data and the same trained encoders. Beating it means a higher rank correlation with true real error across the same pool.

The key ablation removes the filter from the signal. It scores each set by the raw encoder residual magnitude on real data, normalized only by the encoder's own predicted variance and with no temporal propagation. This isolates whether the recursion is the ingredient that carries information. Recursion contributes through innovations that combine the process model's prediction, and through the whiteness term.

Our method counts as winning if its Kendall rank correlation with true real error is significantly above both label-free competitors across bootstrap resamples of the pool, and within a small margin of the few-sequence labeled oracle.

Why not just…?

Why not just keep a small labeled real validation set? Often that is the right answer, and it is our oracle. The case for innovations rests on settings where real labels require motion capture or survey-grade localization while unlabeled streams are free. It also rests on how noisy a two- or three-sequence labeled estimate is. If the few-sequence oracle already matches the full oracle's ranking, our method's practical value shrinks to the fully label-free case. We will report this plainly.

Why not just rescale the learned covariances on real data, a single temperature fit after training? This is a real threat to any claim about fixing miscalibration, and it is why this proposal does not claim to fix it. A global scale can repair the mean of $\epsilon_t$. It cannot repair temporal correlation or input-dependent errors in the learned noise model, and it says nothing about which synthetic set to train on. We will run temperature-rescaled versions of every filter. If rescaling collapses the differences in real error between sets, then our score is mostly measuring noise scale. In that case we would expect the whiteness term to carry the residual ranking signal, and we will report its separate contribution.

Why not just use a black-box recurrent network trained on synthetic data and pick data by its validation loss? A recurrent network has no innovation covariance, so it offers no consistency test. The file reports that such models drift badly: 25.7 m versus 14.3 m on KITTI (How to Train Your Differentiable Filter), and 60.3 cm versus 33.3 cm RMSE on House3D (Particle Filter Networks). The structured filter is therefore both the better estimator and the only one that exposes this diagnostic.

Why not just align the simulator with COMPASS first? COMPASS needs real trajectories paired with simulator rollouts and targets tens of physical parameters. It is complementary: $C(D)$ could serve as its objective when labels are absent. We test this in a small final experiment that tunes one or two generator noise parameters by minimizing $C(D)$.

Proposed Experiments and Decisive Checks

The first testbed is fully controlled. Disc tracking with occlusion, following How to Train Your Differentiable Filter, serves as source. The "real" domain is a held-out simulator variant with heavy-tailed noise, bias, and drift, where every injected error is known. This is where the blind-spot map is built.

The second testbed is visual-inertial odometry. We train a dEKF and a BEVO-style filter on CARLA and TartanAir sets that vary in noise model, weather, and motion diversity. We then compute $C(D)$ on KITTI sequences treated as unlabeled, and true error on KITTI sequences 09–10.

The third testbed is optional and depends on data access. It uses depth-based articulated tracking, with Isaac Sim training and RBO evaluation, in the style of Deep Measurement Updates for Bayes Filters, with the filter made end-to-end differentiable. House3D cannot serve as a real-world test because it has no real counterpart.

Within the KITTI testbed we also estimate an exchange rate for recursive filters. We find how many synthetic sequences match the real error of a filter trained on a given number of real sequences. We then check whether $C(D)$ recovers this curve without labels. The result can be set against the 30:1 figure from On the Equivalency, Substitutability, and Flexibility of Synthetic Data.

Three checks decide the outcome. The first is the rank correlation of $C(D)$ with true real error on KITTI across at least twenty synthetic sets, against the distribution-distance baseline and the no-recursion ablation. The second is whether the controlled testbed confirms the predicted blind spot: insensitivity to absorbable encoder bias, with sensitivity restored when a second sensor is added. The third is whether the ranking survives temperature rescaling. If the first check fails, the idea is wrong. If the second shows no blind spot, our theory of what the score measures is wrong even if the score works. If the third fails, the score is a noise-scale detector and should be presented as only that.