Re-deriving the State Encoder After Sim-to-Real Dynamics Adaptation

Many learned models of physical systems come in two parts. The first is a dynamics model, which predicts how the hidden state of a machine evolves. The second is an encoder, which reads a short window of recent inputs and measured outputs and returns the current hidden state. This works because, for an observable system, that window pins the state down uniquely. When such a model is pretrained in a simulator and moved to real hardware, practitioners must decide which part to retrain on the scarce real data. Robotics papers disagree on the answer. One line of work freezes the simulated dynamics and retrains the encoder. Another freezes the encoder and retrains only the dynamics, and reports that unfreezing the encoder destroys performance.

This note argues that for state-space models the disagreement rests on a misunderstanding. The correct encoder is not an independent component. It is mathematically the inverse of the dynamics and output maps over the look-back window. If the real dynamics differ from simulation, the correct encoder differs too. So neither freezing choice is right. We propose to adapt only the dynamics on real data, as a small physics-anchored residual, and then re-derive the encoder. The adapted model is used as a new simulator, and it generates unlimited synthetic windows with known ground-truth states, broad excitation and realistic sensor corruption. The encoder is then retrained on those windows instead of being fine-tuned on a few real ones. The expected payoff is greatest in the two places where existing encoder methods break: very small real datasets, and test inputs outside the range of the training data.

Background and the gap

The encoder family here is the subspace encoder, SUBNET. In this approach a feed-forward network approximates the reconstructability map, the function from a past input-output window to the current state. It is trained jointly with the state transition network $f_\theta$ and the output map $h_\theta$, using a prediction loss over short, overlapping trajectory segments (Nonlinear state-space identification using deep encoder networks; Deep subspace encoders for nonlinear system identification). The approach extends to several settings:

All of this work fits models on data from a single source. Two studies make the source-mismatch problem concrete. The LFR study was validated only on a simulated toy problem. The one hardware study, on an autonomous Nissan Leaf, trained from track data alone with no simulator prior (Identification of the nonlinear steering dynamics of an autonomous vehicle). Several papers also report sharp error spikes once the system leaves the operating region covered by the training data, because bounded activations saturate.

The sim-to-real literature supplies the motivating tension. In simDP, a simulation-trained action decoder is frozen and a new observation encoder is trained on a few real demonstrations (Encoder-Adapted Sim-to-Real Transfer of Simulation-Trained Diffusion Policies for Robot Manipulation). SimDist does the reverse: it freezes the encoder and updates only latent dynamics, and finds that unfreezing the encoder causes complete performance loss (Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation). Neither paper models an observable state space, so neither can say why one choice works in its setting.

Two further results constrain the design.

Core Claim

Fix the latent coordinates to the simulator's physical states, plus a few extra learned states. Once that is done, the reconstructability map is determined by the dynamics and output maps. The best adaptation recipe then depends on the kind of sim-to-real gap:

  • Pure sensor gaps (bias, latency, noise statistics) call for adapting the encoder's input path. The dynamics stay as they are, which is the simDP-style choice.
  • Dynamics gaps call for adapting the dynamics, as in SimDist. Freezing the encoder then leaves it inconsistent with the new dynamics.
  • The fix is to regenerate the encoder from the adapted model by training it on synthetic rollouts.

We claim this regenerated encoder will beat both freezing choices and full fine-tuning when real data is scarce and when test signals extrapolate beyond the real training range. The reason is that the adapted model can be excited far more broadly than the real experiment was.

The encoder's staleness can be measured directly on unlabeled real data with a consistency residual:

$$r = \frac{1}{N}\sum_k \big\| \psi(w_{k+1}) - f\big(\psi(w_k), u_k\big) \big\|^2 .$$

Here $w_k$ is the past input-output window ending at time $k$, $\psi$ is the encoder, and $f$ is the adapted transition map. A large $r$ means the encoder and the dynamics disagree about what the state is. We predict that $r$ grows with the size of the dynamics gap under a frozen encoder and shrinks after re-derivation.

Research Program

Stage one: simulation pretraining. Each benchmark gets a nominal physics simulator. We train a grey-box SUBNET in the LFR form of Learning-based model augmentation with LFRs on domain-randomized rollouts:

  • physical parameters are randomized;
  • injected sensor noise, delay and offsets are randomized;
  • in simulation the encoder receives a supervised state loss on the physical-state coordinates, alongside the usual truncated prediction loss.

This anchoring removes the freedom to reshuffle latent coordinates, and that freedom is what otherwise makes "freeze the encoder" an ambiguous statement.

Stage two: dynamics adaptation on real data. We use a small real training set, sweeping 1%, 5%, 20% and 100% of each benchmark's training data. Only the neural residual block and a small sensor-correction layer are trained, with the truncated prediction loss. The encoder stays fixed during this step.

Stage three: encoder re-derivation. The adapted model is driven with broad randomized inputs, including amplitudes beyond the real training set. Its outputs are passed through a sensor-corruption model whose parameters are fitted to residuals on the real data. The encoder is retrained on these synthetic windows with supervised state targets, plus the real-data prediction loss and the consistency residual $r$. Stages two and three alternate two or three times.

Stage four: diagnosis. We repeat the whole program under controlled gap types to test whether the recipe-depends-on-gap-type claim holds. Gaps are sensor-only (latency, bias), parametric dynamics (mass, friction coefficient) or structural dynamics (an unmodeled mode or overflow). Encoder state error is reported wherever ground-truth states exist.

Baselines

Naive references. The do-nothing reference is the simulation-pretrained model deployed zero-shot on real data. It shows how large the gap is. The from-scratch reference is SUBNET trained only on the same real subset, which is the method of Nonlinear state-space identification using deep encoder networks. At full data this is the strongest existing published method on these benchmarks. The file reports:

We rerun it with the published architecture and the same tuning budget as our method. The proposal is only useful if it matches these numbers at 100% data and beats them at small data or on the Silverbox extrapolation split.

Best-linear-approximation initialization. SUBNET initialized from the system's best linear approximation (BLA, the best linear model fitted to the data) (Initialization Approach for Nonlinear State-Space Identification via the Subspace Encoder Approach) is the strongest competing way to give SUBNET a prior from real data alone. The file gives no exact test value for it; it states only that this initialization "achieved the lowest simulation NRMS error on the test set." We rerun it on identical subsets.

Grey-box augmentation on real data alone. LFR augmentation fitted on real data without simulation pretraining isolates whether the simulator rollouts add anything beyond the physics structure. On its 3-DOF testbed the file reports RMSE 0.00529, against 0.00557 for black-box SUBNET.

Freezing choices. Three adaptation strategies start from the same simulation-pretrained model and are given the same gradient steps and hyperparameter search budget:

  • full fine-tuning of all parts;
  • a SimDist-style recipe that freezes the encoder and adapts dynamics;
  • a simDP-style recipe that freezes the dynamics and retrains the encoder.

Tuned physics with a filter. TuneNet-style residual parameter tuning of the simulator, followed by an extended Kalman filter for state estimation, represents the classical route.

Key ablation. Stage three is run without its synthetic rollouts: the encoder is retrained only on real windows plus the consistency loss. This isolates the contribution of using the adapted model as a simulator.

Why not just…?

Why not just fine-tune the whole simulation-pretrained model on the real data? This is the most serious threat. With enough real data it may close the gap on its own, since the encoder sees real windows directly. Its weakness is the regime the file flags: encoders fail outside the range of their training windows, and a few real runs cover little of that range. The data-fraction sweep and the Silverbox extrapolation split decide this. If full fine-tuning matches re-derivation at 1–5% data and on extrapolation, the central claim is wrong and we will report that.

Why not just randomize the simulator more widely and deploy zero-shot? The theory in Understanding Domain Randomization for Sim-to-real Transfer requires reality to lie inside the randomized family. The structural gaps used here fall outside any parameter setting of the nominal simulators, for example tank overflow absent from a pure Bernoulli model, or an unmodeled mass. Our zero-shot reference, trained with wide randomization, measures how far this alone gets.

Why not just tune simulator parameters and run a Kalman filter? For purely parametric gaps this may be competitive, and that outcome would be informative. TuneNet, however, can only adjust parameters that already exist in the simulator. A filter on top of it adds iterative cost at every step, whereas the encoder observer in Computationally efficient predictive control based on ANN state-space models avoids this, with LPV-MPC solver latency of 4.79 ms against 16.30 ms for IPOPT. The controlled gap-type study shows on which gap types this baseline holds up.

Proposed Experiments

Benchmarks and simulators. Each real benchmark is paired with a simulator that has a known gap:

  • Silverbox: a Duffing simulator with misspecified coefficients.
  • Wiener-Hammerstein: an idealized block model with a simple diode static nonlinearity.
  • Cascade tank: a Bernoulli model with no overflow.
  • Coupled electric drive and electro-mechanical positioning system: models with only viscous friction.

For the three continuous-time benchmarks (cascade tank, coupled electric drive, positioning system), the file says SUBNET "matched or outperformed" specialized methods and trains in roughly 15 minutes. It gives no per-dataset figure; no reported result was found on Lacuna for those numbers, so we rerun the baselines. If the authors can share the Nissan Leaf data, a linear bicycle model serves as its simulator.

Controlled testbed. Here the simulator is the deliberately deficient 2-DOF linear model and "reality" is the 3-DOF hardening system from Learning-based model augmentation with LFRs. Synthetic latency, offsets and non-stationary noise are added on top. Because true states are available, this is where encoder state error and the residual $r$ are validated.

Metrics. We report:

  • test simulation RMS and NRMS error;
  • extrapolation error on the Silverbox extrapolated split;
  • encoder state error where ground truth exists;
  • the consistency residual $r$;
  • real-data fraction needed to reach the from-scratch full-data error.

Decisive checks. The program has three decisive checks.

  1. Re-derivation must beat full fine-tuning and both freezing recipes at small real-data fractions and on extrapolation. It must not lose at full data.
  2. The ranking of the freezing recipes should follow the gap type: the encoder-retraining recipe wins on sensor-only gaps, and the dynamics-retraining recipe beats it on dynamics gaps.
  3. Removing the synthetic rollouts from stage three should erase most of the gain. If it does not, the benefit comes from the consistency loss, and the proposal should be reframed around that loss.