Checkpoint Reload Explains Most of the Measured State Gap in Two State Discovery Reproductions

Weijia Han, Lisha QuarXiv 2026 · arXiv

TL;DR

Researchers have discovered that the "state gap"—a discrepancy between latent states stored during training and those reconstructed at inference—in the World Machine model is primarily caused by a simple procedural mismatch: reloading a "best" checkpoint. By matching the weights used for inference to those that wrote the training state table, they eliminated over 90% of this inconsistency, revealing that common fixes like distillation and iterative refinement are surprisingly ineffective at closing the remaining gap.

What Problem Does This Paper Address?

Latent world models are designed to compress long histories of observations into compact "states" used for prediction and control. A major challenge arises in models like World Machine, which uses a procedure called State Discovery (SD).

During training, World Machine maintains a "state table" that is updated in parallel every epoch. however, when it comes time for inference (using the model in the real world), that table isn't available. The model must instead perform a "cold start," reconstructing states through a sequential rollout from an all-zero initialization.

The authors investigate a fundamental consistency question: Do the states reconstructed at inference actually match the states the model learned during training? If there is a significant "state gap," the model's performance might be built on a shaky, inconsistent foundation.

Methods

The authors performed a rigorous "reproduction study" using Toy1D, a synthetic benchmark. They focused on three specific populations of states to isolate the source of the gap:

  1. Stored Table: States written during the final training pass using the last epoch's weights ($\theta_{last}$).
  2. Reloaded Rollout: States generated by a sequential rollout using the "best" validation checkpoint ($\theta_{best}$), which is standard practice.
  3. Matched Rollout: States generated by a sequential rollout using the same weights that wrote the table ($\theta_{last}$).

To measure the gap, they used the 1-Wasserstein distance ($W_1$) and linear probes. They also tested two common strategies to see if they could bridge the residual gap without needing the original state table:

  • Causal Distillation: Training a Temporal Convolutional Network (TCN) to map observations directly to stored states.
  • Iterative Refinement: Applying "frozen" SD updates to the inference states to nudge them closer to the training distribution.

Datasets

The study utilizes the Toy1D benchmark, a synthetic environment where observations are generated from a 3D damped linear system.

  • Configuration: 10,000 sequences of 1000 steps.
  • Split: 60% training, 20% validation (8,000 items), and 20% testing.
  • Task: The model must decode a specific latent dimension from noisy, 2-channel linear readouts.

Key Insights and Ideas

  • The "Checkpoint Reload" Culprit: The most significant "aha moment" is that the state gap isn't necessarily a failure of the model's architecture or the SD algorithm itself. Instead, it's a side effect of the training pipeline. Because the "best" checkpoint is reloaded after the final state table is written by the "last" weights, the two are fundamentally out of sync.
  • Cold Start as a Fixed Point: The authors prove that a sequential rollout is actually a fixed point of the State Discovery update rule. This means that simply running more updates on a cold-started state won't change it—it’s already "stable," even if it’s different from the training state.
  • Distillation is Not a Silver Bullet: Even with wide receptive fields, a distilled TCN student could not meet the success criteria (a 50% reduction in the gap). This suggests the mapping from observations to latent states is more complex than simple amortization can capture.

Key Results

The findings highlight that procedural consistency is more important than algorithmic refinement for this specific problem:

  • Weight Matching: Simply using $\theta_{last}$ for the rollout instead of $\theta_{best}$ removed 90% to 91% of the measured state gap (measured by mean per-dimension $W_1$ distance).
  • Error Reduction: Item-by-item RMS error dropped from 0.36 to 0.14 for one seed and 0.43 to 0.16 for another when weights were matched.
  • Visualizing the Gap: As seen in Figure 1, the gap is most pronounced at the very beginning of a sequence (position 1) and decays over time, but matching the weights (the blue lines) keeps the error significantly lower across the entire sequence.
Figure 1: Per-position W1, stored table versus cold-start rollout (log axis, positions 1 to 199), seed 42 solid, seed 7 dashed, under θbest (red) and θlast (blue); labels: the seed-42 position-1 value and the seed-42 means over positions 30 to 199.
  • Failure of Refinement: Neither causal distillation nor frozen SD updates met the pre-specified success criteria (50% gap reduction). In fact, as shown in Figure 2, while updates can move states closer to the rollout, they often fail to move them significantly closer to the stored table under the reloaded checkpoint.
Figure 2: Frozen SD-update trajectories for seed 42 under (a) reloaded θbest and (b) table-writing θlast. Solid: from the table; dashed: from cold start; gold: 50 percent threshold. By K=10, (a) approaches 0.03; no comparable floor is observed in (b) through K=10 (seed 7 and definitions: Appendix C).

Limitations and Future Work

The authors are transparent about several limitations:

  • Scope: The study is limited to two seeds on a single synthetic benchmark (Toy1D). It remains to be seen if "checkpoint reload" accounts for a similar proportion of the gap in complex, real-world datasets (like video or robotics).
  • The Residual 10%: While 90% of the gap is explained, the mechanism behind the remaining 10% of the inconsistency remains unknown.
  • Downstream Impact: The paper measures the state gap but does not explicitly measure how much this gap hurts final performance (e.g., control accuracy or prediction error) in a deployed system.

Future research could investigate whether training procedures that explicitly account for the checkpoint reload (perhaps by updating the table after selecting the best checkpoint) could eliminate this discrepancy entirely.

Summary updated

Abstract

Temporal models that maintain per-sequence latent states during training must reconstruct those states when the stored table is unavailable at inference. This creates a basic consistency question: do the reconstructed states agree with those formed during training? We study this question for State Discovery in World Machine, which updates stored states in parallel each epoch but performs inference by sequential rollout from an all-zero start. On Toy1D, we analyze two paired-checkpoint runs whose decoded-state MSE is 27 to 36 percent above the published value. We compare the stored table with rollouts under the best-validation checkpoint reloaded after training and the last-epoch weights that wrote the table. Matching the rollout to the table-writing weights removes 90 to 91 percent of the mean per-dimension Wasserstein-1 distance. Causal distillation and frozen SD updates do not satisfy their pre-specified criteria under either checkpoint. Overall, checkpoint reload accounts for most of the measured gap in these runs.
BibTeX
@misc{han2026checkpoint,
  author = {Weijia Han and Lisha Qu},
  title = {{Checkpoint Reload Explains Most of the Measured State Gap in Two State Discovery Reproductions}},
  year = {2026},
  eprint = {2610.34891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34891},
}
Figure