Abstract
ObjectZero is a novel model-based reinforcement learning (MBRL) framework that integrates structured, object-centric representations with Monte Carlo Tree Search (MCTS). By replacing traditional monolithic latent vectors with discrete "slots" and utilizing Graph Neural Networks (GNNs) for dynamics modeling, ObjectZero achieves superior sample efficiency and reasoning capabilities in complex multi-object robotic environments. Evaluations on robotic manipulation benchmarks demonstrate that ObjectZero converges faster than state-of-the-art monolithic models while maintaining competitive final performance.
Introduction
Traditional model-based reinforcement learning agents, such as MuZero or Dreamer, typically represent the world as a single, undifferentiated latent vector. While effective for many tasks, these monolithic models often struggle to generalize when objects move, interact, or vary in number. Humans, conversely, naturally decompose scenes into discrete entities, allowing for more robust reasoning about physical interactions.
The primary challenge in structured RL is integrating object-level understanding into high-performance planning frameworks without relying on manual annotations or object masks. ObjectZero addresses this by combining unsupervised object discovery with GNN-based dynamics and MCTS, bridging the gap between factorized world models and state-of-the-art planning.
Methodology
ObjectZero builds upon the architecture of EfficientZeroV2, introducing several key object-aware modifications to the core reinforcement learning pipeline.
1. Slots Extractor
Instead of a standard CNN encoder, ObjectZero utilizes a pretrained, frozen encoder (SLATE for synthetic scenes or DINOSAUR for realistic ones). These models employ "Slot Attention" to group visual features into a set of $K$ vectors, or "slots," each representing a potential object in the scene. This allows the agent to leverage a stable, pretrained visual vocabulary.
2. GNN Dynamics and Heads
To predict state transitions, ObjectZero uses a Graph Neural Network (GNN). Each slot is treated as a node, and interactions are modeled via an edge model, allowing the agent to explicitly reason about how objects (e.g., a robot gripper and a target block) affect one another. Similarly, the reward, value, and policy functions are GNN-based, aggregating information across all slots to inform decision-making.
3. Planning with MCTS
The agent utilizes the learned GNN dynamics model to perform look-ahead simulations during MCTS. By simulating future trajectories at the object level, the agent can identify optimal action sequences based on predicted object interactions.
Experimental Results
The model was evaluated on two robotic manipulation benchmarks: CausalWorld (Object Reaching), involving a tri-finger robot, and Robosuite (Block Lifting), featuring a Panda robotic arm.
Key Findings
- Faster Convergence: In the CausalWorld reaching task, ObjectZero demonstrated significantly faster convergence than all baselines, including the monolithic EfficientZeroV2. This suggests that object-centric priors allow for more efficient exploration and learning.
- Competitive Performance: In the visually complex Robosuite block-lifting task, ObjectZero outperformed other object-centric baselines (such as ROCA and OCRL) and achieved parity with the state-of-the-art monolithic EfficientZeroV2.
- Interaction Modeling: The $O(K^2)$ complexity of the GNN allows the model to explicitly account for collisions and causal effects, which is a significant advancement over models that treat objects as independent entities.
Discussion and Future Work
ObjectZero demonstrates that structured, factorized world models can be successfully integrated into state-of-the-art planning frameworks. A key insight is that frozen, pretrained encoders are sufficient for object discovery, removing the need for simultaneous learning of perception and action.
However, limitations remain. The quadratic scaling of fully connected GNNs may become a bottleneck in scenes with many objects. Furthermore, the model's performance is inherently tied to the quality of the unsupervised object discovery. Future research will explore joint training of the encoder and world model, as well as scaling the system to higher-dimensional, more crowded environments.
Summary updated