TL;DR
This paper presents CHEF, a novel iterative "collect-and-infer" reinforcement learning framework that enables robots to learn complex manipulation tasks directly from real-world experience, drastically improving data efficiency. CHEF achieves near-perfect performance on the challenging RGB Stacking benchmark, outperforming prior state-of-the-art methods by leveraging off-policy learning and a multi-stage data collection and policy optimization process.
What Problem Does This Paper Address?
Learning complex robotic manipulation tasks using reinforcement learning (RL) directly on real robots has historically been hindered by the immense amount of data required. This data inefficiency makes it impractical for most real-world applications. While simulation-to-real (sim2real) approaches and learning from demonstrations (LfD) have shown promise, they have their own limitations: sim2real relies on accurate simulators which are hard to build, and LfD is capped by human performance and requires expert data. This paper tackles the challenge of making RL on real robots data-efficient and robust, without relying on simulators or expert demonstrations, by effectively reusing all collected experience, even sub-optimal data.
Methods
The core of the paper is the CHEF (Collect-and-Infer) framework, an iterative approach inspired by the "Collect-and-Infer" paradigm. It consists of four stages:
- Collect (Stage 1): An initial phase of online, off-policy RL using the SAC-Q multi-task algorithm. This stage aims to collect a diverse dataset by training policies for multiple sub-tasks, ensuring broad state-action space coverage. The goal is to gather as much experience as possible, even if the policies are sub-optimal.
- Hyperparameter Exploration (Stage 2): An offline RL phase where multiple candidate policies are trained using the data collected in Stage 1. This stage is crucial for exploring various hyperparameters (e.g., network architectures, learning rates) and algorithm settings without needing further real-world interaction.
- Execute (Stage 3): The best policies identified in Stage 2 are deployed on the real robot. This stage collects additional, higher-quality data specifically for the main task, and importantly, allows for the evaluation of different hyperparameter choices.
- Fine-tune (Stage 4): The final stage involves offline RL to fine-tune the best performing policy using the combined dataset from both Stage 1 and Stage 3. This iterative process allows for continuous improvement and robust policy learning.
Key to CHEF's success is the use of multi-task RL with a shared network torso (especially for the critic) and off-policy learning methods that can effectively utilize diverse and potentially sub-optimal data.
Datasets
The paper uses the RGB Stacking benchmark for evaluation. This benchmark involves stacking five different triplets of colored geometric shapes, requiring precise manipulation and understanding of geometric affordances. The authors note that the contact dynamics of this task are difficult to accurately replicate in simulation. The data used for training and evaluation is entirely real-world robot experience, collected through the CHEF framework itself.
Key Insights and Ideas
- Data Re-use is Key: The most significant insight is that by intelligently reusing all collected data, including sub-optimal trajectories from early learning stages, data efficiency can be dramatically improved. This contrasts with approaches that discard "failed" attempts.
- Iterative Collect-and-Infer: The staged "collect-and-infer" approach allows for efficient exploration of hyperparameters and policy improvements without continuous, costly real-world interaction.
- Multi-task Learning for Data Efficiency: Training with multiple sub-tasks, even if not all are the primary goal, helps create a more diverse and informative dataset, which is crucial for stabilizing offline RL.
- Offline RL's Potential: Contrary to some prior beliefs, the paper demonstrates that modern off-policy RL algorithms, when combined with multi-task learning and good data coverage, can perform well in an offline setting, even on challenging real-world tasks.
- Real-World Sensors Matter: The ability to learn directly from real-world data allows the use of sensors (like wrist force sensors) that are difficult to accurately model in simulation, leading to more robust and nuanced policies.
Key Results
- The CHEF approach achieved a near-perfect success rate of 96% on the RGB Stacking benchmark.
- This performance significantly outperforms prior state-of-the-art methods by 14% (96% vs. 82%).
- A single iteration of offline RL after the initial data collection (Stage 2) was sufficient to outperform baselines by 10% (92% vs. 82%).
- The use of haptic observations (force/torque sensors) improved success rates on a challenging triplet from 43% to 78% after the first offline RL round.
- The fine-tuning stage (Stage 4) achieved comparable performance to training from scratch but in half the computational time.
Limitations and Future Work
- Task Curriculum: The paper notes that the specific set of sub-tasks used formed a natural curriculum that worked well. Future work could explore how task selection impacts performance, especially with larger models and more diverse domains.
- Iterative Stages: While the presented CHEF schema required only one iteration of the Execute and Fine-tune stages, the authors suggest that repeating these stages could further "hill-climb" to higher performance. The optimal frequency for repeating the Hyperparameter Exploration stage remains an open question.
- Generalization: While the approach is data-efficient, further research could investigate how to generalize learned policies to entirely new tasks or environments.
- Exploration Strategies: The paper touches upon the challenges of exploration for offline RL and suggests further investigation into optimal exploration strategies within the collect-and-infer framework.
Summary updated