Abstract
Accuracy and generalization of dynamics models is key to the success of model-based reinforcement learning (MBRL). As the complexity of tasks increases, so does the sample inefficiency of learning accurate dynamics models. However, many complex tasks also exhibit sparsity in the dynamics, i.e., actions have only a local effect on the system dynamics. In this paper, we exploit this property with a causal invariance perspective in the single-task setting, introducing a new type of state abstraction called model-invariance. Unlike previous forms of state abstractions, a model-invariance state abstraction leverages causal sparsity over state variables. This allows for compositional generalization to unseen states, something that non-factored forms of state abstractions cannot do. We prove that an optimal policy can be learned over this model-invariance state abstraction and show improved generalization in a simple toy domain. Next, we propose a practical method to approximately learn a model-invariant representation for complex domains and validate our approach by showing improved modelling performance over standard maximum likelihood approaches on challenging tasks, such as the MuJoCo-based Humanoid. Finally, within the MBRL setting we show strong performance gains with respect to sample efficiency across a host of other continuous control tasks.
Introduction
Model-Based Reinforcement Learning (MBRL) is prized for its sample efficiency, as it learns a "world model" to plan or simulate data. However, these models often suffer from poor generalization. Because a model is typically trained on data from a specific policy, it often fails when the agent explores new, unseen parts of the state space.
The authors identify spurious correlations as a primary culprit. Standard models attempt to predict the next state using the entire current state and action. In complex systems, many variables are actually independent (e.g., moving a left arm doesn't directly change the position of a right foot). Standard maximum likelihood estimation (MLE) often picks up on coincidental patterns between these unrelated variables, leading to models that "break" when the agent's behavior changes.
Methods: Leveraging Causal Sparsity
The paper proposes a framework to leverage causal sparsity—the idea that each dimension of the next state only depends on a small subset of current state variables, its "causal parents."
Model-Invariant Abstractions
The authors define a per-variable state abstraction that preserves transition probabilities. They prove that an abstraction consisting only of the causal parents of a variable is model-invariant and allows for optimal policy learning. This enables compositional generalization: by learning factored, per-variable dynamics, the model can generalize to novel compositions of state variables it has never seen together before.
Practical Implementation
Since finding exact causal parents in neural networks is difficult, the authors introduce a self-supervised Invariance Loss for deep RL. This loss (based on KL divergence of prediction similarities) penalizes models for relying on features that aren't consistently necessary for prediction, effectively forcing them to ignore spurious correlations. This invariant learner is integrated into the SAC-SVG algorithm, a state-of-the-art actor-critic framework.
Experimental Results
The approach was evaluated on standard continuous control benchmarks, including MuJoCo environments like Cheetah, Ant, and the particularly challenging Humanoid-v2.
Key Insights
- The Humanoid Breakthrough: Humanoid tasks are notoriously difficult for MBRL due to high dimensionality. The authors show that spurious correlations are a major reason standard models fail here. By masking out irrelevant dimensions (simulating causal knowledge), model error drops significantly.
- Superior Generalization: In Humanoid-v2, the invariant learner achieved significantly lower test error than standard MLE, especially in low-data regimes (200k to 500k samples).
- Sample Efficiency: When integrated into an RL agent, the method showed strong performance gains. For example, on the Ant and Humanoid tasks, their method (Ours H-3) outperformed or matched state-of-the-art algorithms like PETS and POPLIN.
Limitations and Future Work
While promising, much of the formal theory relies on linear structural equation models, while the practical implementation uses non-linear neural networks as a proxy. Furthermore, the current work focuses on proprioceptive states (joint angles, velocities). Applying this to high-dimensional pixel data remains a challenge, as it requires first learning a compressed, sparse latent space. Finally, the authors suggest that explicitly enforcing sparsity constraints could lead to even tighter error bounds in future iterations.
Summary updated