When a physics simulator is fitted to real sensor data and its predictions still miss, two very different things can be happening. The simulator's parameters may be drifting or mistuned, such as a friction coefficient or a ventilation rate that changed. Or the simulator's equations may simply be missing some physics. The first problem is fixed by re-estimating parameters. The second needs a new model term, and no amount of parameter tuning will fix it. Current practice cannot cheaply tell these apart. On a real underground farm, a particle filter tracked parameter shifts about 30 times faster than full Bayesian calibration, yet could not separate drift from structural model error. The slow Gaussian-process calibration was still needed for that (Ward et al.).
We propose to answer the question with a quantity that fixed-lag particle smoothers already compute: the per-time-step gradient of the log-likelihood with respect to the model parameters, obtained without differentiating through resampling. If the model is correctly specified, classical likelihood theory says these per-step score contributions must behave like unpredictable noise. They should have zero mean given the past and no autocorrelation, and their spread should equal the curvature of the log-likelihood. When the model's equations are wrong, these properties break in a specific, measurable way that parameter drift does not produce. The same stream of score contributions therefore works as a cheap, online, directional test. It says whether the simulator is structurally wrong, and it says which parameters are absorbing the error. We would validate the test on real physical data, not only on synthetic "realities."
Core Claim
A particle filter approximates the distribution of a hidden state with a cloud of weighted samples. Fisher's identity turns the log-likelihood gradient (the "score") into an average, over smoothed trajectories, of gradients of the transition and observation densities. Truncating the smoother to a window of $L$ past steps keeps the cost linear in the number of particles. Differencing the running fixed-lag score between consecutive time steps gives a per-step score contribution $s_t(\theta) \approx \nabla_\theta \log p(y_t \mid y_{1:t-1}, \theta)$, where $y_t$ is the observation at time $t$ and $\theta$ the simulator parameters. The same smoother also gives a per-step curvature $H_t(\theta) \approx \nabla^2_\theta \log p(y_t \mid y_{1:t-1}, \theta)$. If the model is correctly specified at the true parameter, $s_t$ is a martingale difference: it has zero mean given the past and is serially uncorrelated. The information-matrix equality also holds:
$$ \mathbb{E}\left[s_t s_t^\top + H_t\right] = 0 . $$
Under pure parameter drift, an online estimator that re-adapts $\theta$ restores both properties once it has caught up. The only symptom is a transient nonzero mean in $s_t$. Under structural misspecification the picture is different. At the best-fitting "pseudo-true" parameter, the scores still average to zero over the whole record, because that is what "best-fitting" means. However, the increments become autocorrelated, and the matrix $D_T = \frac{1}{T}\sum_t (s_t s_t^\top + H_t)$ stays away from zero no matter how the parameters are tuned.
The claim is that three signals, taken together, separate drift from missing physics at the cost of a single filter pass. The first is a portmanteau autocorrelation statistic on $s_t$. The second is the information-matrix discrepancy $D_T$. The third is the sensitivity of the score estimate to the lag $L$. On the third signal: under a correct, fast-forgetting model the estimate stabilizes quickly as $L$ grows, while long-memory structural error keeps shifting it. Because $s_t$ is a vector over parameters, the rows of $D_T$ with large entries point to the parameter directions that are covering for the missing term.
How This Relates to Prior Work
The estimator machinery is borrowed. PF-SEFI establishes fixed-lag Fisher-identity scores at linear cost, and on real Waymo LiDAR tracks it reaches an average displacement error of 0.275 m against 0.643 m for a differentiable particle filter. It uses the score only for maximum-likelihood fitting under fixed Ackermann kinematics, and it never asks whether that kinematic model is wrong. The second-order particle MCMC work (Dahlin et al., PMH with gradient and Hessian) shows that Hessians come out of fixed-lag smoothing almost for free. We reuse this to build $H_t$. Linear-time backward smoothers, namely backward importance sampling and Parisian particle Gibbs, supply lower-bias alternatives to plain fixed-lag truncation. We use them as an ablation that checks whether the test's power comes from the misspecification signal rather than from truncation artifacts.
Three papers motivate the gap. Ward et al. found that particle filters conflate drift with model bias. BAD-PODS builds Fisher-identity gradients into online experimental design but assumes the model is exactly right and tests only on synthetic systems. Kersting et al. model numerical solver error but explicitly not structural error. Two papers constrain the design. DA-SHRED recovers missing terms such as the Gray-Scott $V^2$ term with deterministic recurrent decoders and sparse regression, offline, without posterior uncertainty. Our test does not name the missing term. It only says that one exists and which parameters it contaminates, so a library-based discovery step remains complementary. Collins et al. show that derivative-free tuning of physics engines closes 14% to 91% of the trajectory gap but costs 279 to 662 hours. Their residual error after tuning is exactly what our test should classify as structural or not.
Research Program
The first component is the statistic and its null distribution. Fixed-lag truncation and finite particle counts bias $s_t$ and $H_t$, so the textbook chi-square nulls do not apply. We therefore use simulation to calibrate the test. After fitting $\theta$ to real data, we simulate many datasets from the fitted simulator, run the identical estimator with the same $L$ and particle count, and take the empirical distribution of the autocorrelation statistic and of the norm of $D_T$ as the null. In this setup the simulator is the null hypothesis and reality is the alternative. A small reported p-value then means "real data are distinguishable from the simulator at its best-fitting parameters, in these specific parameter directions." We will also derive how the truncation bias in $s_t$ scales with $L$ under geometric forgetting, and use that to pick the smallest lag at which the null distribution stops depending on $L$.
The second component is a streaming version. We compute the statistics over a sliding window alongside an online score-ascent update of $\theta$, so that drift appears as a transient mean shift that the update absorbs, while structural error appears as persistent autocorrelation. The third component is a repair loop. When the test flags a problem, we add a residual term to the transition density: a small Gaussian-process or neural correction, gated to the flagged state components. We then check whether the test passes afterwards. This turns the diagnostic into a stopping rule for model expansion.
Baselines
The naive reference is to accept whatever the calibrated simulator predicts and treat residual error as noise. On the classification task this means always predicting "drift," and its score is the base rate of our labeled scenarios.
The strongest standard diagnostic is innovation-based model checking. It runs the same particle filter, computes probability integral transforms of one-step-ahead predictions, and applies uniformity and autocorrelation tests to them. It uses exactly the same filter, particles, lag and simulation-calibrated null as our method, so the only difference is whether the diagnostic lives in observation space or in parameter-score space. Beating it means higher accuracy at separating drift from structural error at matched false-alarm rate, and correct identification of the contaminated parameter group, which a scalar innovation test cannot provide.
The strongest existing method for detecting model bias is static Kennedy-O'Hagan calibration with a Gaussian-process discrepancy term, as run by Ward et al.. They report predictive RMSE of 0.9% for it against 1.1% for the particle filter, and a runtime of 22,572 s against 786 s on a 40-point dataset. We rerun it on the same data windows with the same Gaussian-process emulator. Beating it means matching its bias detection, meaning a nonzero discrepancy when and only when structure is missing, at particle-filter cost.
For simulator tuning on the Kinova data, the reference is the differential-evolution tuning of Collins et al., with a reported 14% to 91% improvement over default settings. The comparison asks whether our score-guided tuning reaches similar trajectory error with far fewer rollouts, and whether our test correctly flags the tasks where tuning left a structural gap. No reported result was found on Lacuna for any method on drift-versus-bias classification itself, so these numbers are new.
Two ablations isolate the key ingredients. The first replaces the fixed-lag smoother with backward importance sampling or Parisian particle Gibbs to remove truncation bias. The second drops the information-matrix term and keeps only the autocorrelation term.
Why not just…?
Why not just test the one-step prediction residuals, which every filter already produces? This is the most serious threat. Innovation tests detect misspecification of every kind, but they respond to a drift-induced mean shift and to structural error in similar ways. They also say nothing about which parameters are compensating. Our argument for the score test is that the information-matrix equality is violated only by structural error, and violated in specific parameter directions. That argument is unproven. If the innovation baseline matches our classification accuracy in the experiments below, the defensible contribution shrinks to parameter attribution, and we would report it that way.
Why not just fit a Kennedy-O'Hagan discrepancy model? It works, and Ward et al. show it identifies bias, but it costs roughly 30 times more and runs on static batches. Our test is meant to be the cheap online prefilter that decides when that expense is warranted. The experiment is whether its flags agree with a nonzero Kennedy-O'Hagan discrepancy.
Why not just fit an expanded model with a residual term and compare held-out likelihood? Model comparison requires choosing and fitting the expansion first, and it does not localize which physics is missing. We run this comparison as part of the repair loop. If blanket expansion everywhere always wins on held-out likelihood at similar cost, the test adds little beyond interpretability. We would check that directly.
Why not just increase the lag? Longer lags increase path degeneracy and the variance of the gradient. More importantly, the lag-sensitivity curve is one of our signals, so "just increase the lag" is part of the test rather than a replacement for it.
Proposed Experiments
Controlled scenarios come first, because they have ground truth. The first uses the Growing Underground thermal model from Ward et al., which includes their synthetic 50% ventilation drop as a pure-drift case. We add a structural case in which data are generated with an extra moisture-exchange term that the fitted model omits. The second uses Gray-Scott with the $V^2$ term deleted from the fitted model, following DA-SHRED, and varies the term's magnitude to trace detection power curves. In both, the decisive check is the classification error between drift and structure at 5% false-alarm rate against the innovation baseline, plus attribution accuracy for the parameter group.
The real-data experiments use settings where a structural gap is documented but its form is not given. The first is the Kinova benchmark (Collins et al.), with the physics engine as the transition model and motion capture as observations. Simulators there peak at 0.69 m/s against 1.36 m/s for the real arm. The prediction is that our test flags a structural gap on the non-prehensile rolling tasks even after tuning, and attributes it to the friction and maximum-velocity parameters. The second is the Hudson Bay lynx-hare series used by backward importance sampling, under a stochastic Lotka-Volterra model. The third is Waymo tracks under PF-SEFI's Ackermann model, where tire slip at high curvature should produce flagged autocorrelation concentrated in the steering parameters.
The decisive check is falsifiable in both directions. After the repair loop adds a residual term gated to the flagged components, held-out predictive likelihood must improve and the test must pass. Adding the same residual to unflagged components must not help. If neither of these holds, the attributions are not trustworthy.
References
- Continuous calibration of a digital twin: Comparison of particle filter and Bayesian calibration approaches
- Particle-Based Score Estimation for State Space Model Learning in Autonomous Driving
- Second-order particle MCMC for Bayesian parameter inference
- Particle Metropolis-Hastings using gradient and Hessian information
- Backward Importance Sampling for Online Estimation of State Space Models
- Particle-based, Rapid Incremental Smoother Meets Particle Gibbs
- Online Bayesian Experimental Design for Partially Observed Dynamical Systems
- Differentiable Likelihoods for Fast Inversion of 'Likelihood-Free' Dynamical Systems
- Data assimilation and discrepancy modeling with shallow recurrent decoders
- Traversing the Reality Gap via Simulator Tuning
- Benchmarking Simulated Robotic Manipulation Through a Real World Dataset