Abstract: We address the problem of effectively composing skills to solve sparse-reward tasks in the real world. Given a set of parameterized skills (such as exerting a force or doing a top grasp at a location), our goal is to learn policies that invoke these skills to efficiently solve such tasks. Our insight is that for many tasks, the learning process can be decomposed into learning a state-independent task schema (a sequence of skills to execute) and a policy to choose the parameterizations of the skills in a state-dependent manner. For such tasks, we show that explicitly modeling the schema's state-independence can yield significant improvements in sample efficiency for model-free reinforcement learning algorithms. Furthermore, these schemas can be transferred to solve related tasks, by simply re-learning the parameterizations with which the skills are invoked. We find that doing so enables learning to solve sparse-reward tasks on real-world robotic systems very efficiently. We validate our approach experimentally over a suite of robotic bimanual manipulation tasks, both in simulation and on real hardware.
Introduction
Bimanual manipulation—using two robotic arms in coordination—represents a "holy grail" for robotics, offering the versatility required for human-like tasks. However, automating these motions is notoriously difficult. Traditional Reinforcement Learning (RL) faces three primary hurdles in this domain: high-dimensional action spaces, hybrid action spaces (deciding both which discrete skill to use and its continuous parameters), and sparse rewards, where a robot only receives feedback upon successful task completion.
Standard end-to-end RL attempts to map raw sensory input directly to these complex actions, a process that is often too slow and sample-inefficient for practical real-world application. This research introduces a method to bridge this gap by separating high-level task logic from low-level execution details.
Methodology: The Task Schema Approach
The core innovation of this work is the disentanglement of the policy into two distinct components:
- Task Schema (State-Independent): This is a fixed sequence of skills (e.g., "Left arm: Top Grasp, Right arm: Side Grasp" followed by "Left arm: Twist"). The researchers hypothesize that for many manipulation tasks, this high-level "recipe" remains constant regardless of the object's specific location.
- Skill Parameterization (State-Dependent): A neural network ($\phi$) processes environmental data (RGB images) to determine specific arguments for the skills, such as precise $(x, y)$ coordinates for a grasp or the required lifting force.
Workflow
The training process follows a three-step pipeline. First, both the schema and parameterization are learned in a low-dimensional simulation where object states are known. Second, the learned high-level schema is "frozen" and transferred to the real world. Finally, the robot performs real-world fine-tuning, learning only the visual parameterization ($\phi'$) from raw RGB images using a Fully Convolutional Network. This allows the robot to focus on "where" to act, having already mastered "what" to do.
Experimental Validation
The approach was tested across four distinct task families involving common household objects and two Sawyer robot arms:
- Lateral Lifting: Coordinating two arms to lift a long bar or a flexible aluminum tray.
- Picking: Lifting a slippery soccer ball that requires simultaneous pressure from both sides.
- Opening: Unscrewing a glass jar or a water bottle.
- Rotating: Operating tools like a T-wrench or a corkscrew.
Key Results and Insights
By baking the state-independence of the task sequence into the architecture, the researchers achieved significant improvements in sample efficiency. In simulation, the Schema approach reached high success rates far faster than baseline RL models.
In real-world tests, the system learned to solve complex bimanual tasks from raw images in just 4 to 10 hours of training. This is a substantial improvement over standard RL, which typically requires weeks of experience for simpler tasks. Furthermore, the use of a fully convolutional architecture for spatial parameters allowed the robot to generalize to different object positions without re-learning the high-level task logic.
Conclusion and Future Work
This work demonstrates that transferring the logic of a task (the schema) rather than just visual features (domain randomization) is a powerful path for sim-to-real transfer. While the current schemas are "open-loop" sequences, future iterations could incorporate recurrent mechanisms like LSTMs to allow for more reactive, closed-loop behavior if a task fails mid-execution. Additionally, the researchers aim to explore inferring these schemas directly from human demonstrations, further reducing the reliance on manual skill libraries.
Summary updated