Abstract
In this work we propose a novel task framework under which a variety of physical reasoning puzzles can be constructed using very simple rules. Under sparse reward settings, most of these tasks can be very challenging for a reinforcement learning agent to learn. We build several simple environments with this task framework in Mujoco and OpenAI gym and attempt to solve them. We are able to solve the environments by designing curricula to guide the agent in learning and using imitation learning methods to transfer knowledge from a simpler environment. This is only a first step for the task framework, and further research on how to solve the harder tasks and transfer knowledge between tasks is needed.
Introduction
A fundamental challenge in Artificial General Intelligence (AGI) is the ability of agents to generalize beyond specific training scenarios. In reinforcement learning (RL), agents often master specific motor skills but fail when environmental constraints or object placements change slightly. This paper introduces "BlockPuzzle," a benchmark designed to bridge the gap between learning motor skills and understanding the underlying physical rules of an environment. The authors argue that true physical problem-solving requires an intuitive understanding of physics and scene constraints rather than simple pattern recognition.
The BlockPuzzle Framework
The researchers developed a task framework using the MuJoCo physics engine and OpenAI Gym, based on the OpenAI Fetch robotics platform. To focus on reasoning rather than complex manipulation, the robot's gripper is locked, and the action space is restricted to four dimensions.
Tasks are defined by a simple color-coded system:
- Blue (Manipulation): The object the robot must move.
- Green (Goal): The target location or object.
- Red (Constraint): Touching Blue to Red results in immediate failure.
- Grey (Neutral): Obstacles that hinder movement but do not trigger failure.
Two primary environments were tested: BlocksTouch-v0 (a 2-block task) and BlocksChoose-v0 (a 3-block task adding a grey distractor block).
Methodology
To address the difficulty of sparse rewards in these environments, the authors employed three main strategies:
- Reinforcement Learning Baselines: Deep Deterministic Policy Gradient (DDPG) and Policy Gradient with Gaussian Distribution (PGGD) were used as core algorithms.
- Curriculum Learning: Agents were initially trained with blocks spawned close to the robot, with the spawning radius gradually increasing as performance improved. This was critical for convergence.
- Imitation Learning (AggreVaTeD): To solve the more complex 3-block tasks, the authors used an "expert" policy trained on the simpler 2-block task to guide the learner, gradually annealing the expert's influence.
Key Results and Insights
The study revealed several critical findings regarding the scalability and generalization of RL agents. Notably, while imitation learning helped in low-resource settings, scaling up computational resources yielded superior results.
- The Impact of Scale: DDPG trained on 20 cores significantly outperformed other methods, including imitation learning. The authors suggest that more parallel workers provide a more diverse data distribution, allowing the critic network to better model obstacle avoidance.
- Functional Generalization: The high-resource DDPG agent achieved a 90% success rate on "Challenge" configurations—unseen scenarios where obstacles were placed directly in the path—demonstrating that it learned to maneuver based on physical rules rather than just memorizing paths.
- The "Distractor" Problem: Adding a single neutral grey block significantly increased task difficulty, as agents initially struggled to distinguish between relevant goals and irrelevant obstacles.
| Method | Normal Success | Challenge Success |
|---|---|---|
| DDPG (2 blocks) | 0.44 | 0.24 |
| DDPG (3 blocks, 20 cores) | 0.97 | 0.90 |
| PGGD + AggreVaTeD | 0.64 | 0.40 |
Conclusion
The BlockPuzzle framework provides a robust testbed for physical reasoning in robotics. While standard RL struggles with generalization, the combination of curriculum learning and scaled-up parallel training allows agents to master complex constraints. Future work will focus on handling higher degrees of freedom, variable rule sets, and more advanced transfer learning across diverse puzzle configurations.
Summary updated