Abstract
Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability. However, existing approaches suffer from structural limitations and optimization difficulties for common environments with multiple dynamic objects. In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP), which employs a three-level learning architecture that enables efficient object-based dynamics learning from raw visual observations. We also design a spatial-temporal relational reasoning mechanism for MAOP to support instance-level dynamics learning and handle partial observability. Our results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments for learning environment models. We also demonstrate that learned dynamics models enable efficient planning in unseen environments, comparable to true environment models. In addition, MAOP learns semantically and visually interpretable disentangled representations.
Introduction
In Model-Based Reinforcement Learning (MBRL), the core challenge is for an agent to learn a "world model"—a predictive simulator of how the environment responds to its actions. A robust world model allows an agent to "imagine" consequences and plan effectively. However, current state-of-the-art models frequently struggle with two primary issues: generalization and complexity. Most models learn pixel-level patterns specific to a training screen, causing them to fail when layouts change. Furthermore, in environments with numerous moving entities like enemies and projectiles, learning object identities and their interactions simultaneously is a computationally grueling task.
This paper introduces the Multi-level Abstraction Object-oriented Predictor (MAOP), a self-supervised framework designed to learn object-oriented dynamics that are independent of specific layouts. By decomposing raw video into distinct objects and their interactions, MAOP enables agents to understand the underlying "rules" of a brand-new environment layout without retraining.
Methods: Multi-Level Abstraction
The core innovation of MAOP is its three-level hierarchy, where simpler levels of abstraction guide the learning of more complex ones. This structured approach avoids the optimization pitfalls of trying to learn representations and dynamics from scratch simultaneously.
- Motion Detection Level: This coarse, unsupervised step identifies "blobs" of movement, providing initial proposals for where dynamic objects might be located.
- Dynamic Instance Segmentation Level: This level refines the initial proposals into distinct object masks. It utilizes a "discrepancy loss" based on the principle that pixels belonging to the same rigid object should exhibit consistent affine transformations.
- Dynamics Learning Level: The most fine-grained level, which includes an Object Detector and Instance Localization for tracking. The Dynamics Net serves as the system's brain, using a Relation Net to calculate inter-object effects (e.g., collisions) and an Inertia Net to account for an object's own history. This inertia component is crucial for handling gravity or partial occlusions. Simultaneously, a Background Constructor learns to represent static environment elements separately.
Experimental Evaluation
The researchers evaluated MAOP on several challenging benchmarks. The primary test for zero-shot generalization was Monster Kong, a complex custom game where a monster breathes fire across varying layouts. Additionally, MAOP was tested on a suite of six Atari Games (including Skiing, MsPacman, and Breakout) and environments like Flappy Bird, which feature moving cameras and high object counts.
Results and Discussion
MAOP demonstrates extreme sample efficiency and superior generalization compared to baselines like CDNA and traditional pixel-based AC models. On the Monster Kong benchmark, MAOP achieved 90% accuracy with only 3,000 samples, whereas CDNA required 300,000 samples and still underperformed.
In terms of generalization, MAOP maintained nearly perfect prediction accuracy (0.99 agent accuracy) when moved to entirely unseen level layouts. In contrast, traditional models saw their performance crater. The practical utility of these learned models was demonstrated by plugging MAOP into a Monte Carlo Tree Search (MCTS) planner. In unseen environments, the MCTS + MAOP agent performed almost as well as an agent using the actual game engine (Reward: 38.19 vs 38.41), far outperforming standard DQN agents (Reward: 13.67).
Conclusion
MAOP represents a significant advancement in self-supervised dynamics learning by treating hierarchy as a curriculum. By focusing on "what moves together" before "how things interact," the model learns physics-based rules rather than memorizing frames. While the current model assumes rigid bodies and requires a pre-set number of object classes, its ability to handle partial observability and generalize to new environments marks a major step toward robust, interpretable world models in complex 2D domains.
Summary updated