Learning to Scaffold the Development of Robotic Manipulation Skills

Lin Shao, Toki Migimatsu, Jeannette Bohg

Abstract

Contact-rich manipulation tasks, such as peg insertion or bolt tightening, present significant challenges for robotic reinforcement learning due to high-dimensional action spaces and sensor noise. This paper proposes a "scaffolding" framework that enables robots to autonomously simplify these tasks by placing physical fixtures in their environment. By employing a hierarchical learning system—an outer loop for fixture placement and an inner loop for skill acquisition—the robot can achieve high success rates up to 8x faster than traditional methods. Furthermore, a curriculum transitioning from physical to virtual constraints allows the robot to retain these skills even after the scaffolds are removed.

Introduction

Learning "contact-rich" manipulation is notoriously difficult for robots. Unlike simple pick-and-place operations, tasks like inserting a peg into a hole or tightening a bolt require fine-grained control and the ability to handle uncertainty from sensors and motor imprecision. While humans naturally use the environment to simplify tasks—such as sliding a key against a lock plate to find the keyhole—most robotic reinforcement learning (RL) agents attempt to learn skills in isolation without modifying their surroundings.

This research argues that robots should instead learn to scaffold their own development. By creating physical constraints that "funnel" uncertainty and guide the robot toward success, the complexity of the learning problem is significantly reduced. This approach mimics the use of training wheels for a child learning to ride a bicycle, providing a safety net that allows for exploration in a more controlled state space.

Methodology: The Two-Loop Scaffolding Framework

The authors propose a hierarchical learning system consisting of two nested loops designed to optimize both the environment and the robot's actions.

The Outer Loop: Fixture Selection

This loop treats the environment as a "Contextual Bandit" problem. Given a depth image of the scene, the system proposes an optimal pose (position and orientation) for a physical fixture. A key challenge here is the discontinuous nature of the reward: moving a fixture by just a millimeter can be the difference between a perfect guide and a total blockage. To address this, the authors introduced the Smoothed Zooming Algorithm, which adaptively discretizes the action space to handle sharp changes in reward and focus on high-performing areas.

The hierarchical system architecture showing the flow from depth-based fixture placement (outer loop) to RGB-based skill learning (inner loop).

The Inner Loop: Skill Learning

Once the fixture is placed, the robot enters the inner loop to learn the actual manipulation task. Using a variant of the Asynchronous Advantage Actor-Critic (A3C) algorithm, the robot processes RGB images and utilizes operational space control to master the skill within the constrained environment.

Physical Curriculum and Virtual Scaffolding

To ensure the robot does not become overly dependent on the physical fixture, the framework includes a curriculum-based "fading" process. The physical fixture is gradually replaced by a virtual potential field—a software-based "soft" constraint. This field is then slowly moved away, forcing the robot to transition from physical guidance to pure visual-motor coordination.

Experimental Evaluation

The system was evaluated in both the PyBullet simulation environment and on real-world hardware, including Franka Panda and Kinova Gen3 arms. The researchers focused on three challenging tasks: round peg insertion, wrench manipulation of a bolt, and shallow-depth insertion of a battery-like object into a mobile phone slot. Domain randomization was employed during simulation to ensure the models could generalize to the physical world despite the "reality gap."

Results and Discussion

The results demonstrate that scaffolding provides a dramatic speedup in learning. In peg insertion tasks, robots using an optimal fixture achieved a 20% success rate in just 5,000 steps, whereas those without a fixture required 40,000 steps to reach the same milestone.

Comparison of learning curves across three stages: learning without a fixture (blue), learning with a physical fixture (orange), and transitioning to a virtual potential field (green).

Beyond speed, the scaffolding approach also raised the performance ceiling. Robots managed nearly 80% success on tasks where they previously plateaued at 20% without environmental assistance. Crucially, the system proved robust; models trained entirely in simulation with fixtures achieved 100% success in real-world transfer for insertion and wrench tasks. The virtual potential field curriculum was essential for this, allowing the robot to maintain a 70% success rate even after all physical help was removed.

Conclusion and Future Work

This work highlights the power of environmental modification in robotic learning. By allowing robots to "help themselves" through physical scaffolding, complex contact-rich tasks become significantly more accessible.

Future research directions include moving beyond predefined fixture geometries to allow robots to design or select their own scaffolds. Additionally, automating the "fading" speed of the curriculum and exploring the use of multiple or dynamic fixtures could further enhance the autonomy and versatility of the system in complex assembly environments.

Summary updated

BibTeX
@inproceedings{shao2020learning,
  author = {Lin Shao and Toki Migimatsu and Jeannette Bohg},
  title = {{Learning to Scaffold the Development of Robotic Manipulation Skills}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  doi = {10.1109/icra40945.2020.9197134},
}
Figure