Abstract
Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability. However, existing approaches suffer from structural limitations and optimization difficulties for common environments with multiple dynamic objects. In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP), which employs a three-level learning architecture that enables efficient object-based dynamics learning from raw visual observations. We also design a spatial-temporal relational reasoning mechanism for MAOP to support instance-level dynamics learning and handle partial observability. Our results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments for learning environment models. We also demonstrate that learned dynamics models enable efficient planning in unseen environments, comparable to true environment models. In addition, MAOP learns semantically and visually interpretable disentangled representations.
1. Introduction
In model-based Deep Reinforcement Learning (DRL), an agent's ability to plan effectively depends on its "world model"—a representation of how the environment evolves in response to actions. Traditional pixel-based models attempt to predict future video frames as a whole, which often leads to poor generalization in environments with varying layouts and high computational complexity when multiple dynamic objects are present.
Humans perceive the world not as a grid of pixels, but as a collection of interacting objects. To bridge this gap, the Multi-level Abstraction Object-oriented Predictor (MAOP) is proposed. MAOP is designed to disentangle objects from backgrounds and learn their underlying physics, enabling zero-shot generalization to novel environment configurations.
2. The MAOP Framework
MAOP utilizes a three-level abstraction hierarchy to simplify the complex task of learning dynamics from raw pixels:
- Motion Detection Level: This level identifies moving "blobs" using unsupervised techniques, providing coarse-grained proposals for object locations.
- Dynamic Instance Segmentation Level: Using the motion proposals, this level learns to group pixels into distinct, rigid object masks through an "Instance Splitter" and "Merging Net."
- Dynamics Learning Level: This fine-grained level predicts future object positions using two specialized components: the Relation Net, which models spatial interactions (e.g., collisions), and the Inertia Net, which handles temporal effects and partial observability by considering object history.
A Background Constructor maintains a representation of static elements, allowing the model to generate predicted frames by overlaying dynamic object masks onto the learned background.
3. Experimental Evaluation and Results
The researchers evaluated MAOP on several benchmarks, including a customized Monster Kong game and standard Atari titles. A key focus was "zero-shot generalization," where models were trained on one layout and tested on multiple unseen environments.
3.1 Sample Efficiency and Accuracy
MAOP demonstrates remarkable sample efficiency. In Monster Kong, it achieved 90% 0-error accuracy in unseen environments using significantly fewer samples than baselines. As shown in Figure 3, MAOP (purple line) rapidly outperforms models like CDNA and OODP, which struggle to achieve high accuracy even with more data. The initial flat period in the MAOP curve reflects the hierarchical "proposal" phase where motion and segmentation are established before dynamics learning begins.
3.2 Planning and Interpretability
When integrated with a Monte Carlo Tree Search (MCTS) planner, the MAOP-learned model performed comparably to the "true" game engine in unseen environments. Furthermore, the model's reliance on explicit object masks provides inherent interpretability; the system can automatically identify the "agent" (the object controlled by actions) with nearly 100% accuracy across different games.
4. Discussion and Conclusion
MAOP represents a significant step toward interpretable and generalizable world models. By leveraging a hierarchical abstraction, it avoids the optimization pitfalls of joint segmentation and dynamics learning. While currently focused on 2D rigid objects, future work will explore extensions to 3D environments and more complex visual changes.
Reference: Zhu, G., Wang, J., Ren, Z., Lin, Z., & Zhang, C. (2020). Object-Oriented Dynamics Learning through Multi-Level Abstraction. Proceedings of the AAAI Conference on Artificial Intelligence.
Summary updated