When Does Dynamics Randomization Test Context Inference?

Oleg Smirnov, Sofiane Ennadir, John Pertoft, Bjartur Hjaltason, Sara KarimiarXiv 2026 · arXiv

TL;DR

Dynamics randomization is the standard way to test if a robot or agent can "infer context" (like wind or weight), but this paper shows that many benchmarks are easier than they look. The authors introduce predictive headroom—a metric that reveals how much information the current state already "leaks" about the hidden dynamics, often leaving very little for a dedicated context-encoder to actually do.

What Problem Does This Paper Address?

In model-based Reinforcement Learning (RL), we often use "Contextual MDPs" where physical parameters (like the mass of an object or the strength of a gust of wind) change between episodes. To succeed, an agent should ideally infer these hidden parameters from its experience.

However, the authors argue that simply randomizing parameters doesn't guarantee a good test of context inference. Two things often happen:

  1. Implicit Identification: The current state might already contain the "evidence." For example, if a constant wind is blowing, your current velocity already reflects that wind. A simple model can use the state to predict the next move without ever "thinking" about the wind explicitly.
  2. Weak Kernel Differences: The agent might only visit parts of the environment where the randomized parameters don't actually change the outcome much (e.g., wind doesn't matter if you are walking into a wall).

The gap is that we lack a formal way to measure how much "room for improvement" actually exists for a context-aware model.

Methods

The authors formalize a metric called Predictive Headroom ($I_{emp}$). This is the expected one-step log-loss difference between an optimal predictor that knows the hidden context and one that doesn't.

To understand why headroom might be low, they decompose the information into three stages:

  • Structural Variation ($I_{struct}$): How much the dynamics differ across the entire theoretical state space.
  • Matched-Query Variation ($I_{match}$): How much the dynamics differ only in the states the agent actually visits.
  • Visited Headroom ($I_{emp}$): How much information is left after accounting for the fact that the visited states might already reveal the context.

By comparing these, researchers can tell if a benchmark is "easy" because the dynamics are too similar (low $I_{struct}$) or because the agent is accidentally "cheating" by reading the context from its current state (large drop from $I_{match}$ to $I_{emp}$).

Datasets

The researchers evaluated these metrics on three controlled environments:

  • MiniGrid-Wind: A discrete grid-world where a "drift" (wind) is sampled each episode.
  • MiniGrid-Resistance: A grid-world where actions randomly fail with a certain probability.
  • Point-Mass-Attractor: A continuous 2D system where one of four hidden corners exerts a force on the agent.
Figure 2: Point-Mass-Attractor force fields. The first four panels show the context-dependent force −k(p −θi) for each kernel Tθi. The final panel shows the prior-averaged force −kp.

Key Insights and Ideas

  • The State is a Snitch: In many environments, the state acts as a "memory" of past transitions. In the Attractor environment, the agent's velocity is a direct consequence of the hidden force. Even a "context-blind" model can perform well because the evidence has accumulated in the state.
  • Visitation Matters: If an agent spends all its time in a corner where the randomized dynamics are blocked by a wall, the "randomization" is effectively zero.
  • Resets Reveal the Truth: The authors found that if they "reset" the agent's state (e.g., teleporting it to a random spot) while keeping the same episode context, the predictive headroom spikes. This proves that the agent was previously using its state to "implicitly identify" the context.

Key Results

The study shows a massive discrepancy between "theoretical" randomization and "actual" predictive headroom.

Figure 1: Dynamics variation and predictive headroom across perturbation strengths. (a,b) Wind and Resistance mechanisms. (c–e) The structural reference Istruct and estimated visited headroom bIemp differ substantially on Wind and Attractor but remain close on Resistance. Attractor’s structural curve uses the interior Gaussian reference. Shading shows bootstrap intervals.

As seen in the table below, for the Wind environment, the structural variation is high (0.871), but the actual visited headroom is tiny (0.077). This means 91% of the "contextual information" is either irrelevant to the agent's path or already visible in the current state.

Table 1: Comparisons at representative settings. Fixed and matched columns retain prior context weights, while visited headroom uses fitted visitation-dependent weights. Attractor’s identical prior references use its query-invariant interior kernel. Uncertainties are the larger one-sided deviations from the 95% bootstrap intervals.

Limitations and Future Work

  • One-Step Focus: The paper focuses on one-step prediction. While this is fundamental for world models, context might have different value for long-term planning or multi-step reasoning.
  • Estimation Complexity: Calculating these metrics requires either knowing the ground-truth transition kernels or having very high-quality learned models.
  • Future Directions: The authors suggest that future RL benchmarks should report "Predictive Headroom" as a standard metric. This would allow researchers to see if a small gain from a new context-aware algorithm is due to a bad algorithm or simply because the benchmark had no "headroom" to begin with.

Summary updated

Abstract

Environments with randomized dynamics are widely used to ask whether world models need explicit context inference. However, randomization can leave little predictive information for a context encoder to add, because distinct latent parameters may produce similar transitions at visited states, or those states may already reveal the active context. We formalize the remaining one-step information as predictive headroom and compare kernels under fixed and visited state--action distributions to distinguish changes in query coverage from changes in context weights. We analyze discrete and continuous environments and find low headroom both when kernels differ little and when visitation reduces an otherwise large discrepancy. At the same time, resetting state components increases headroom, showing that context evidence accumulates in state. We propose reporting this headroom alongside the randomized parameters, so that gains from explicit context inference can be interpreted relative to the information missing from the current state and action.
BibTeX
@misc{smirnov2026when,
  author = {Oleg Smirnov and Sofiane Ennadir and John Pertoft and Bjartur Hjaltason and Sara Karimi},
  title = {{When Does Dynamics Randomization Test Context Inference?}},
  year = {2026},
  eprint = {2610.32414},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32414},
}
Figure