TL;DR
This paper introduces SAP, a novel method for zero-shot policy learning that decomposes sparse, terminal rewards into spatially and temporally fine-grained rewards. By learning these decomposed rewards and a dynamics model from prior experience, SAP enables agents to generalize to new, unseen environments without any further interaction or fine-tuning, outperforming existing methods on challenging tasks like Super Mario Bros and robotic manipulation.
What Problem Does This Paper Address?
A major hurdle in AI is enabling agents to learn in one environment and then generalize to new, unseen environments without extensive retraining or additional data collection. This is particularly challenging when the agent only receives sparse, delayed rewards (e.g., only at the end of a game level). Current methods often struggle with this "zero-shot generalization" because they tend to memorize specific solutions rather than learning transferable underlying principles. Humans, on the other hand, can leverage prior knowledge and an understanding of object properties and their interactions to adapt quickly to new situations. This paper aims to bridge this gap by mimicking this human-like generalization capability.
Methods
The core of the proposed method, SAP (Scoring, Aggregation, Planning), involves three key components:
- Spatial Temporal Reward Decomposition (Scoring Network): The agent learns a "scoring network" that takes a "contingency-aware observation" (a localized view around the agent) and an action, and outputs a reward for that specific time step and spatial region. This observation is further divided into smaller "sub-regions" to capture finer-grained compositional information. The network is trained by aggregating these per-step rewards to match the original sparse terminal reward.
- Dynamics Model Learning: A neural network is trained to predict the next state given the current state and action. This is learned in a self-supervised manner from the same prior experience data.
- Model Predictive Control (MPC) for Planning: With the learned scoring function (acting as a reward function) and the dynamics model, the agent uses MPC to plan an optimal action sequence in the new, unseen environment. This allows the agent to act without direct interaction or reward signals in the test environment.
What's novel is the combination of learning these fine-grained, spatially decomposed rewards and using them within an MPC framework for zero-shot generalization, moving beyond simple state-action mapping or generic RL.
Datasets
The paper evaluates SAP on two main domains:
- Super Mario Bros: The original game environment is used, with observations preprocessed into 84x84 grayscale stacked frames. The task involves surviving and moving as far right as possible, with a delayed terminal sparse reward.
- Blocked Reacher (3D Robotics Task): A 7-DoF robotics arm environment where the goal is to reach a specific point while avoiding block obstacles. The environment is discretized into voxels, and the observation space is continuous.
The data used for training SAP consists of "exploratory trajectories" collected from suboptimal policies in a training environment. This data is crucial as it's the only source of information the agent has before being tested in a new environment.
Key Insights and Ideas
- Decomposition is Key for Generalization: Decomposing sparse rewards into finer spatial and temporal components is crucial for enabling agents to generalize to new environments. Simply learning a global reward function is insufficient.
- Compositionality Matters: Breaking down observations into smaller, meaningful sub-regions allows the agent to learn about fundamental building blocks of the environment, which are more likely to be present and reusable in unseen scenarios.
- Human Priors as Inspiration: The method draws inspiration from human learning, where we leverage an understanding of object properties and their interactions to adapt to new tasks. SAP attempts to replicate this by learning "contingency-aware" observations.
- Model-Based Planning for Zero-Shot: Combining learned reward functions with a learned dynamics model within a planning framework (MPC) is a powerful strategy for zero-shot generalization, as it allows for proactive decision-making without needing direct environmental feedback.
Key Results
- Super Mario Bros: SAP significantly outperforms baselines like Behavioral Cloning, DARLA, and RUDDER in zero-shot generalization tests. For instance, on the "World 2 Stage 1" (new task), SAP achieves a return of 790.1, while RUDDER achieves 587.9 and BC is much lower. Ablation studies show that both spatial and temporal decomposition contribute significantly to this performance.
- 3D Robotics Task: SAP also demonstrates superior performance in the Blocked Reacher environment, requiring significantly fewer steps to reach the goal compared to MBHP and Rudder in both training and testing configurations. The learned negative scores for crashing into obstacles are highlighted as a key factor in its efficiency.
- Qualitative Results: Visualizations show that the learned scores are meaningful and align with human intuition (e.g., avoiding "koopa" enemies in Mario). The agent's learned behavior in new levels is often reasonable, demonstrating its ability to generalize.
Limitations and Future Work
- Limitations:
- The method relies on having prior "exploratory" data from a training environment. The quality and coverage of this data can impact performance.
- While the method is designed to be robust to errors in the learned dynamics model, significant inaccuracies could still hinder performance.
- The "Piranha Plants" in Super Mario Bros were not recognized because they did not appear in the training data, indicating that generalization is limited to components present in the training experience.
- Future Work:
- The paper suggests further exploring the use of SAP as an unsupervised method for object discovery due to the learned decomposed rewards.
- Investigating more complex environments and tasks, including those with more dynamic or abstract object interactions, would be a natural next step.
- Exploring different aggregation functions for rewards and more sophisticated planning horizons could also yield improvements.
Summary updated