Model-Invariant State Abstractions for Model-Based Reinforcement Learning

Manan Tomar, Amy Zhang, Roberto Calandra, Matthew E. Taylor, Joelle Pineau

TL;DR

This paper introduces model-invariance, a per-variable state abstraction that exploits local sparsity in transition dynamics to eliminate spurious correlations in model-based reinforcement learning (MBRL). By learning dynamics models that enforce causal consistency across individual state dimensions, the method achieves significantly better generalization in low-data regimes and improves sample efficiency on challenging MuJoCo benchmarks like Humanoid.

What Problem Does This Paper Address?

In Model-Based Reinforcement Learning (MBRL), an agent learns a transition model of its environment to plan actions or generate synthetic rollouts. However, learned dynamics models frequently overfit and fail to generalize when a shifting policy navigates into previously unvisited areas of the state space.

Standard maximum-likelihood estimation (MLE) models often latch onto spurious correlations—for example, treating a robot's shoulder movement as predictive of its knee joint dynamics simply because they co-occurred in the training data. While physical systems typically exhibit sparse dynamics (each next-state dimension depends only on a local subset of current variables), traditional state abstraction methods (such as bisimulation) evaluate state transitions holistically rather than factorizing per state dimension.

Methods

The authors formalize state abstraction on a per-variable level and connect it to causal invariance:

  • Model-Invariant State Abstractions: Rather than grouping entire states, the abstraction $\phi_i$ is defined per state variable $x^i$, ensuring that abstract states preserve the transition probability $P(x^i_{t+1} \mid x_t, a_t)$. The authors prove that conditioning strictly on the causal parents $\text{PA}(x^i_{t+1})$ yields an optimal invariant abstraction with bounded value loss.
  • Linear Setting (Certainty Equivalence): Using Invariant Causal Prediction (ICP), the model isolates the causal parents of each state component across data generated by multiple policies (treated as causal environments).
  • Practical Non-Linear Algorithm (Deep MBRL):
  • To avoid combinatorial parent searches, the authors adapt the self-supervised ReLIC framework.
  • Two (or more) dynamics heads ($f$ and $h$) share an underlying bottleneck representation $\phi$.
  • Models are trained with standard one-step MLE loss augmented by an invariance loss (a symmetric KL divergence over cosine similarities scored by a projection critic $g$). Crucially, this similarity is evaluated on a randomly sampled single state variable at each step rather than the full state vector, discouraging the network from using global spurious features to predict local variables.
  • Integrated directly into the SAC-SVG (Stochastic Value Gradients) policy optimization framework.
       ┌───────────┐ ──> Model f ──> [x̂_t+1] ──┐
(s, a) │ Shared φ  │                           ├──> Critic g (per variable i) ──> Invariance Loss
       └───────────┘ ──> Model h ──> [x̂_t+1] ──┘

Datasets

The authors evaluated their framework on:

  • Synthetic 3-Variable Linear MDP: A toy environment where each state variable strictly depends only on its own previous value, tested under policy-induced distribution shifts.
  • MuJoCo Continuous Control Benchmarks: Standard Gym/MuJoCo benchmarks including Humanoid-v2, HalfCheetah, Walker2d, Hopper, and Ant, evaluated using protocols from MBPO and POPLIN.

Key Insights and Ideas

  • Per-Dimension Invariance vs. Per-State Invariance: Enforcing invariance across the entire state vector fails to eliminate cross-variable spurious correlations. The authors empirically demonstrate that applying the invariance loss per individual state dimension is essential for improved generalization.
  • Bypassing Explicit Causal Graph Discovery: Finding the exact causal directed acyclic graph (DAG) in high-dimensional continuous control is computationally prohibitive. Using an auxiliary self-supervised consistency loss between model augmentations serves as an effective, end-to-end differentiable proxy for causal sparsity.
  • Higher Horizons Benefit More: As the imagination/rollout horizon ($H$) in MBRL expands, compounding model errors usually degrade performance. By removing spurious correlations, the invariant model exhibits larger relative gains over baseline MLE models at longer rollout horizons.

Key Results

  • Linear MDP: Invariant prediction converged stably to the true dynamics with only 500 samples (error $2.7 \times 10^{-3}$), whereas MLE retained an error of $14.5 \times 10^{-3}$ due to distribution shifts across policy environments.
  • Humanoid-v2 Dynamics Modeling: In the 200k–500k sample regime, model-invariance significantly reduced test error compared to standard MLE. In addition, representations learned with model-invariance exhibited lower cross-feature correlation ($1.25 \times 10^3$ vs. $1.63 \times 10^3$), confirming induced latent sparsity.
  • Policy Performance at 200k Steps:
  • POPLIN Benchmark: Model-invariant SAC-SVG outperformed all baselines (PETS, POPLIN-A, POPLIN-P, METRPO, SAC) on 3 out of 4 tasks (e.g., Cheetah: 7067 vs. POPLIN-P's 4235; Ant: 3124 vs. POPLIN-P's 2330).
  • MBPO Benchmark: Outperformed standard SAC-SVG across multiple horizons on Humanoid (e.g., at $H=4$, 3506 vs. 2404) and Ant (4565 vs. 4470).

Limitations and Future Work

  • States vs. Pixels: The current formulation assumes access to structured, factored state variables. Adapting it to raw pixel inputs requires jointly learning to disentangle visual scenes into distinct semantic entities before enforcing model invariance.
  • Theoretical Bounds: The current value bounds do not incorporate an explicit sparsity term (such as maximum causal in-degree), which could yield tighter theoretical sample complexity guarantees.
  • Walker Performance: Like many baseline algorithms, the method struggled to achieve strong positive scores on Walker2d under the POPLIN benchmark setup.
  • Alternative Invariance Objectives: Exploring offline RL extensions using other invariant objectives, such as the AND mask, remains an open direction.

Summary updated

Abstract

Accuracy and generalization of dynamics models is key to the success of model-based reinforcement learning (MBRL). As the complexity of tasks increases, learning accurate dynamics models becomes increasingly sample inefficient. However, many complex tasks also exhibit sparsity in dynamics, i.e., actions have only a local effect on the system dynamics. In this paper, we exploit this property with a causal invariance perspective in the single-task setting, introducing a new type of state abstraction called \textit{model-invariance}. Unlike previous forms of state abstractions, a model-invariance state abstraction leverages causal sparsity over state variables. This allows for compositional generalization to unseen states, something that non-factored forms of state abstractions cannot do. We prove that an optimal policy can be learned over this model-invariance state abstraction and show improved generalization in a simple toy domain. Next, we propose a practical method to approximately learn a model-invariant representation for complex domains and validate our approach by showing improved modelling performance over standard maximum likelihood approaches on challenging tasks, such as the MuJoCo-based Humanoid. Finally, within the MBRL setting we show strong performance gains with respect to sample efficiency across a host of continuous control tasks.
BibTeX
@inproceedings{tomar2022model,
  author = {Manan Tomar and Amy Zhang and Roberto Calandra and Matthew E. Taylor and Joelle Pineau},
  title = {{Model-Invariant State Abstractions for Model-Based Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=BM7RjuhAK7W},
}

Loading related content...

Figure