Considering Context: When World Models Need Context Encoders

Oleg Smirnov, Sofiane Ennadir, John Pertoft, Bjartur Hjaltason, Sara KarimiarXiv 2026 · arXiv

In model-based reinforcement learning (RL), we often deal with environments where the underlying rules—the "context"—change. Maybe the floor is suddenly slippery, or a robot is carrying a heavier load. The standard reflex in the research community is to bolt on a context encoder: a specialized module designed to infer these hidden parameters from experience.

But do we always need them? A new paper from researchers at King AI Labs and Microsoft Gaming argues that we might be over-engineering. They introduce a framework called Predictive Sufficiency to determine exactly when a world model needs a dedicated context encoder and when the agent’s own behavior already does the heavy lifting.

TL;DR

This paper formalizes "predictive headroom"—the potential gain in prediction accuracy from knowing the true environment context. It demonstrates that in many common RL benchmarks, an agent’s own trajectories implicitly identify the context, making dedicated encoders redundant and suggesting that performance gains often come from better optimization rather than "missing" information.

What Problem Does This Paper Address?

Generalization in RL is hard because dynamics vary. These variations are often modeled as Contextual MDPs, where a latent variable $\theta$ (like friction or mass) governs transitions.

The prevailing assumption is that an agent cannot recover this latent context from its own experience, so we must provide it externally or infer it via complex meta-learning or latent encoders. However, the authors point out a subtle truth: an agent that acts in a world also performs inference about it. If a robot moves and drifts to the left, it has already "sensed" the wind. If the world model can see that drift in the history, does it really need a separate "wind encoder"?

The paper seeks to quantify this "headroom"—the gap between what a standard model knows and what an "Oracle" (who sees the true context) knows—and determine if that gap is actually bridgeable by better architecture or if it has already been closed by the agent's own behavior.

Methods

The authors introduce several key theoretical tools:

  1. Predictive Headroom ($I_{emp}$): This is the exact reduction in one-step prediction error (log-loss) you get by revealing the true context $\theta$ to a model.
  2. Headroom Decomposition: They split headroom into three parts:
  3. Information already recoverable from history.
  4. A residual that truly requires the ground-truth context.
  5. The finite-model deficit (errors caused by the model being too small or poorly optimized, rather than lacking information).
  6. Implicit Identification: A phenomenon where the agent’s "visitation distribution" (where it goes and what it does) naturally concentrates the posterior over the context.

To test these, they use "probes"—simple MLP or recurrent models—to measure how much information about the next state is contained in the current state, the history, or the ground-truth context.

Figure 1: Contextual variation in MiniGrid, where a hidden wind variable probabilistically displaces the agent after each move, and walls block the displacement. (a) A model trained on pooled expe- riences with different winds learns the averaged kernel, which has no net displacement and inflated spread and governs no individual episode. (b) Under the same policy the trajectories separate mid- episode, so the state alone is consistent with only one wind and identifies it implicitly, rendering contextual mechanisms partially redundant. (c) Under one and the same context, the trajectory of an agent crossing the open grid (blue) bends downwind and betrays the context, while an agent moving along the downwind border (orange) shows no drift at all, since the obstacle blocks it. What remains for context mechanisms to gain depends on what the agent’s own trajectories already reveal, and on what they cover.

Datasets

The researchers evaluated their framework across a spectrum of difficulty:

  • Controlled Diagnostics: Modified versions of MiniGrid (Wind and Resistance variants) and a Point-Mass-Attractor environment where transition kernels are known analytically.
  • Benchmark Scale: CARL DMC Walker, where they compared an "Identifiable" setting (horizontal wind) vs. a "Hard" setting (actuator strength scaling).
  • Non-i.i.d. Drift: RWRL Walker, where environment parameters evolve over time according to a Markov chain, testing the model's ability to carry information across episode boundaries.

Key Insights and Ideas

  • The "Aha!" Moment on Implicit ID: In many tasks, the state itself "betrays" the context. As shown in Figure 1, if an agent is in a "Wind" environment, its trajectory naturally bends. If the world model sees the state, it effectively sees the context. In these cases, the "predictive headroom" collapses to zero.
  • Headroom $\neq$ MDP Class: You can't tell if a task needs a context encoder just by looking at the environment definition. The same task might have massive headroom under a random policy but zero headroom under a competent policy that naturally explores informative states.
  • Context vs. History: If headroom persists but a model with access to history reaches the same performance as one with ground-truth context, then a dedicated context encoder is unnecessary—a standard recurrent world model (like DreamerV3) is theoretically sufficient.
  • The Finite Model Trap: Often, adding a context encoder improves performance not because it provides new info, but because it provides an "inductive bias" that helps a finite neural network optimize better.

Key Results

  • Identifiable vs. Hard: In the "Identifiable" Walker task (Wind), headroom was negligible ($\approx 0$). Despite this, some context-aware methods still showed higher returns, implying their benefit was in optimization/learning speed, not in providing missing information.
  • The Power of Velocity: In continuous systems like the Point-Mass-Attractor, the authors found that velocity is the primary carrier of context. Resetting an agent's position but keeping its velocity preserved most of the context information, whereas resetting velocity destroyed it.
  • Non-i.i.d. Benefits: In drifting environments, methods that carried state across episode resets (like "Carried State" or LILAC-style agents) significantly outperformed those that started fresh, confirming that the "headroom" in these tasks resides in cross-episode history.

Limitations and Future Work

  • Closed-Loop Dependency: All measurements are properties of the "closed loop." The information available depends on the current policy. The authors deliberately avoid using a "neutral" reference policy because, in the real world, an agent's own behavior is what defines its information environment.
  • Active Sensing: The paper notes that no standard benchmark currently requires "deliberately informative actions" (active sensing) to identify context. Most benchmarks are "passively identifiable."
  • Future Directions: The authors suggest that future work should focus on environments where context only yields to specific, intentional probes by the agent, and on using "counterfactual resets" to better disentangle the effects of visitation.

For those interested in the math, the full paper provides a rigorous decomposition of predictive risk and proofs for value sensitivity to transition errors in Appendix A.

Summary updated

Abstract

Methods for generalization in model-based reinforcement learning typically assume that an agent cannot recover the latent context governing the environment dynamics from its own experience, and therefore supplies it externally. We formalize and test this assumption with \emph{predictive sufficiency}, which quantifies what access to the context adds to next-step prediction under the visitation distribution an agent induces, and separates that quantity into a history-recoverable part, a residual requiring the true context, and the deficit added by a finite model. We classify context-aware algorithms by the predictive risk their conditioning set can target and demonstrate across environments of increasing identification difficulty that the headroom does not follow the MDP class. The same task under different priors leaves predictive headroom in one setting and nothing distinguishable from zero in another, where the agent's behavior implicitly identifies the context and any benefit of such a mechanism cannot be attributed to missing information. Where headroom persists, the learned state exposes it only partially, and adding the true context still lowers the risk. Our contribution is a practical criterion for matching contextual mechanisms to the information available to them, estimated from the ordinary trained agent without a reference policy.
BibTeX
@misc{smirnov2026considering,
  author = {Oleg Smirnov and Sofiane Ennadir and John Pertoft and Bjartur Hjaltason and Sara Karimi},
  title = {{Considering Context: When World Models Need Context Encoders}},
  year = {2026},
  eprint = {2610.06651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.06651},
}
Figure