Reading a Memory Policy's Mind to Grade Its Synthetic Training Data

Robots that must infer things they cannot see, such as a payload's mass, a floor's friction, or where an object went after it was hidden, need memory. A memory-augmented policy, meaning a controller that carries a running internal summary of its past observations and actions (typically a recurrent network such as a GRU or LSTM), can work out these hidden quantities on the fly. These policies are almost always trained on synthetic data: simulators with randomized physics, or imagined rollouts from a learned world model. The practical question is whether a given synthetic training distribution is good enough. Today the only reliable answer is to deploy the policy in the real target environment and watch it, which is expensive and sometimes dangerous. Existing ways to check synthetic data fidelity compare observations frame by frame. They ignore what a memory policy is most sensitive to: whether real histories push its internal summary into states it never saw during training, and whether it recovers once that has happened.

We propose a simple test that needs no real-world rollouts. Take the trained policy's memory module, feed it a small set of logged real trajectories, and measure two things: how far its internal states land from the cloud of states it visited on synthetic data (a "belief-support gap"), and how long a single surprising observation keeps the internal state off course (a "contamination half-life"). Our claim is that these two hidden-state quantities predict real-world transfer of memory policies, and choose among candidate synthetic distributions, checkpoints, and real-versus-synthetic mixing ratios, better than observation-level fidelity metrics or one-step model error do. Put in two sentences: a memory policy fails in reality when reality drives its memory somewhere synthetic data never took it, and we can detect this offline by running the memory on a few real logs. That turns the policy's own internal state into an evaluation tool for its synthetic training data.

Why this gap is real

Three findings motivate the proposal. Understanding Domain Randomization for Sim-to-real Transfer proves that memory is necessary for domain randomization to work, because the history-dependent policy acts as an implicit system identifier. Domain randomization here means training across many simulators with randomly varied physics. The proof's bounds assume that real dynamics lie inside the support of the randomized simulators, and that every state can be recovered from. Our metric targets exactly the case where the first assumption fails. Investigating Memory in Model-Free RL with POPGym Arcade shows empirically that one out-of-distribution observation can corrupt a recurrent hidden state for hundreds of steps, and that credit gets "smeared" across uninformative past events. It studies synthetic noise injections only, and it leaves open whether model-based training changes the picture. Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective reports 24–30% drops at real deployment and 30–50% drops under lighting or camera shifts. It explicitly cannot tell memory failures apart from perception failures, and it leaves its sim-to-real alignment metrics unvalidated.

Two further papers supply the evaluation idea. Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality judges whether a model is "real enough" by asking whether the distance between real and simulated data exceeds the distance between two independent simulations. We borrow this null-reference construction and apply it to hidden states rather than raw data. On the Equivalency, Substitutability, and Flexibility of Synthetic Data finds that behavioral complexity matters more than photorealism for sequential tracking, raising DanceTrack HOTA from 68.8 to 73.5. This suggests that the right fidelity notion for temporal models is a dynamic one, and a hidden-state metric measures dynamics through the model's own lens. A further constraint comes from Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism. Its method, NEUBAY, trains a recurrent policy (built on Linear Recurrent Units) on long imagined rollouts from a world-model ensemble. It still tunes its real/synthetic mixing ratio and learning rates with target-environment evaluation, and it fails on narrow-coverage data such as AntMaze. Offline selection is therefore an unsolved and practically important problem in exactly the systems we study.

Core Claim

The closest existing approaches judge synthetic data either by observation statistics, as in the simulation Turing-test idea or the rendering-fidelity framing, or by task success after deployment. None of them examines the memory state of the downstream policy. We claim that, for history-dependent controllers trained on synthetic data, two quantities computed from a few logged real trajectories predict deployment return and support near-oracle selection without target rollouts. The first is the normalized distance of real-history hidden states from the synthetic hidden-state distribution. The second is the recovery time of the hidden state after a single-frame perturbation. Concretely, let $z_t$ be the policy's memory state at step $t$ and $Z_{\text{syn}}$ a held-out set of memory states from synthetic histories. We define

$$G = \frac{\mathbb{E}_t\, d_k(z_t^{\text{real}}, Z_{\text{syn}})}{\mathbb{E}_t\, d_k(z_t^{\text{syn}'}, Z_{\text{syn}})},$$

where $d_k$ is the mean distance to the $k$ nearest neighbours and $z^{\text{syn}'}$ comes from a second, independent batch of synthetic histories. $G \approx 1$ therefore means "real is no stranger than another synthetic draw," following the reference construction of the BKT simulation paper. The half-life $\tau$ is the number of steps until $\lVert z_t^{\text{pert}} - z_t \rVert$ falls below half its peak after one observation in a real log is replaced with a sample from a different real episode.

One confound needs care: a logged real trajectory was produced by some behavior policy, not by the policy under test. We therefore build $Z_{\text{syn}}$ by running the same behavior policy, or replaying its logged action sequences, in the synthetic source. The comparison then isolates mismatch in dynamics and observations rather than mismatch in actions.

Research Program

The work has three parts. First, we build a controlled "proxy reality" in which the true transfer return is cheap to compute. We train memory policies on randomized simulators and test them on held-out conditions placed either inside or outside the randomized support. The outside conditions include masses and damping beyond the training range, as in the ExORL dynamics shifts used by Foundation Policies with Memory. They also include unmodeled sensor latency and heavy-tailed, biased sensor noise, the kind of hardware quirk that clean-Gaussian simulators omit according to Nonlinear Attitude Estimation for Small UAVs with Low Power Microprocessors. This produces a population of trained policies, each with a known true transfer return. The population spans randomization widths, architectures (GRU, LSTM, LRU, and a frame-stacked MLP), seeds, and checkpoints.

Second, we compute $G$ and $\tau$ for each policy from a budget of $n$ logged proxy-real trajectories. We sweep $n$ from 5 to 200 to find how much real data the diagnostic needs. This is a measurement-side analogue of the equivalency ratio from the synthetic-data substitutability paper.

Third, we use the diagnostics for selection in a setting where target tuning is currently required: NEUBAY's real/imagined mixing ratio and its rollout-truncation threshold on D4RL. Here the logged dataset plays the role of "real" and the ensemble rollouts play the role of "synthetic." The same experiment answers the open question from POPGym Arcade, namely whether long imagined rollouts protect the recurrent state from contamination or let world-model errors accumulate inside it.

Baselines

The naive reference point selects whichever policy scores best on synthetic validation return, which is what practitioners do when they lack real rollouts. Any useful diagnostic must beat this. An "oracle" reference selects by true target return. We report selection regret relative to it, defined as true return lost compared with the oracle's choice.

The strongest competing diagnostic family is model error on real data. For each candidate synthetic source we compute one-step and $H$-step prediction error of the simulator or world model on the same $n$ logged real trajectories. This uses the same data and compute as our metric. It is the most natural thing a reviewer would compute, and it already captures much of the dynamics mismatch. Our method beats it only if it gives higher rank correlation with true return across the policy population and lower selection regret at equal $n$.

The second competitor is observation-level distribution distance, specifically the maximum mean discrepancy (a kernel-based two-sample distance) between real and synthetic observation windows with the two-simulation normalization. It represents the fidelity view taken by the robot benchmarking paper and the BKT simulation paper.

The third competitor is an offline value estimate, such as fitted-Q evaluation, run on the real logs with the same tuning budget as our $k$-nearest-neighbour bandwidth search.

For policy strength we train recurrent Soft Actor-Critic from Recurrent Off-policy Baselines for Memory-based Continuous Control, which matched full-state oracles on most of that paper's partially observed tasks. That paper reports no numeric scores, so no reported result was found on Lacuna for RSAC returns. We also train FB-GRU, for which no numeric ExORL score is reported either. On D4RL the reference numbers are NEUBAY's reported normalized scores of 34.1 on walker-random and 14.4 on hammer-cloned, both from Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism. Those numbers come from target-tuned hyperparameters. Our selection counts as a success if, with offline-only selection, it recovers within a few points of them while synthetic-return selection does not.

Finally, an ablation isolates the key ingredient. We compute the same $G$ on a frame-stacked encoder's features and on the recurrent policy's input-layer features. If those predict transfer as well as the recurrent memory state, then the memory is not what carries the signal.

Why not just…?

Why not just measure the simulator's prediction error on the real logs? This is the main threat. When the model mismatch is purely in dynamics and the policy uses its memory faithfully, $H$-step error may track transfer as well as our metric does. We expect them to come apart in two cases. The first is when a large error affects variables the policy ignores. The second is when a small error is amplified by the recurrent state, the contamination effect documented in POPGym Arcade. The proxy-reality population includes both cases, latency and noise shifts as well as irrelevant-dimension shifts, so the head-to-head rank correlation settles the question. If model error matches us everywhere, we report that the cheaper diagnostic suffices.

Why not just compare observation statistics? Observation-marginal fidelity is known to miss what matters for sequential models. The substitutability paper finds that behavioral complexity, not appearance, drives gains, and the robot benchmark sees 30–50% drops even with high-fidelity rendering. We still run maximum mean discrepancy at matched $n$ rather than assume it loses.

Why not just fine-tune on the small real set and skip diagnosis? Fine-tuning helps once a candidate is chosen. The substitutability paper shows that 20% real data can match full real training. But fine-tuning does not tell you which synthetic distribution, checkpoint, or mixing ratio to start from, which is the unsolved selection problem in NEUBAY. In addition, recurrent policies are fragile to optimize, as POPGym: Benchmarking Partially Observable Reinforcement Learning shows. We will report whether diagnostic-selected starting points also fine-tune better.

Why not just avoid recurrence with frame stacking, so contamination cannot persist? Fixed windows cannot carry information across long gaps. The recurrent baselines paper shows this on reacher-pomdp, where the goal is visible only on the first frame. Our population includes frame-stacked policies, so the study measures directly whether they trade lower contamination for lower in-support return.

Proposed Experiments

The decisive check is the Spearman correlation between each diagnostic and true transfer return across roughly 300 policies. We run it separately for in-support and out-of-support test conditions, on pendulum-p, cartpole-p, and reacher-pomdp from the recurrent baselines suite, on Stateless CartPole from POPGym, and on ExORL Walker and Quadruped with shifted mass and damping. We also report selection regret against the oracle as a function of real-log budget $n$.

Our prediction is that $G$ will separate out-of-support conditions where model error is small but transfer collapses, and that $\tau$ will add predictive power under latency and noise shifts. On D4RL we select NEUBAY's mixing ratio and truncation threshold offline using $G$ and $\tau$. We compare the true scores against target-tuned NEUBAY and against synthetic-return selection. We also report whether $\tau$ grows with imagined rollout length, which directly answers whether model-based training worsens recurrent contamination. If time permits, a single physical check on a memory-dependent manipulation task in the style of the T4 tasks from the robot benchmarking paper would test whether $G$ ranks a handful of sim-trained policies in the same order as their hardware success rates.

The claim fails if model-error baselines match our rank correlation and regret everywhere, or if the frame-stacked and input-layer ablations predict transfer as well as the recurrent state.