Abstract
Contact-rich manipulation tasks, such as peg insertion or bolt tightening, present significant challenges for robotic reinforcement learning due to high-dimensional action spaces and sensor noise. This paper proposes a "scaffolding" framework that enables robots to autonomously simplify these tasks by placing physical fixtures in their environment. By employing a hierarchical learning system—an outer loop for fixture placement and an inner loop for skill acquisition—the robot can achieve high success rates up to 8x faster than traditional methods. Furthermore, a curriculum transitioning from physical to virtual constraints allows the robot to retain these skills even after the scaffolds are removed.
Introduction
Learning "contact-rich" manipulation is notoriously difficult for robots. Unlike simple pick-and-place operations, tasks like inserting a peg into a hole or tightening a bolt require fine-grained control and the ability to handle uncertainty from sensors and motor imprecision. While humans naturally use the environment to simplify tasks—such as sliding a key against a lock plate to find the keyhole—most robotic reinforcement learning (RL) agents attempt to learn skills in isolation without modifying their surroundings.
This research argues that robots should instead learn to scaffold their own development. By creating physical constraints that "funnel" uncertainty and guide the robot toward success, the complexity of the learning problem is significantly reduced. This approach mimics the use of training wheels for a child learning to ride a bicycle, providing a safety net that allows for exploration in a more controlled state space.
Methodology: The Two-Loop Scaffolding Framework
The authors propose a hierarchical learning system consisting of two nested loops designed to optimize both the environment and the robot's actions.
The Outer Loop: Fixture Selection
This loop treats the environment as a "Contextual Bandit" problem. Given a depth image of the scene, the system proposes an optimal pose (position and orientation) for a physical fixture. A key challenge here is the discontinuous nature of the reward: moving a fixture by just a millimeter can be the difference between a perfect guide and a total blockage. To address this, the authors introduced the Smoothed Zooming Algorithm, which adaptively discretizes the action space to handle sharp changes in reward and focus on high-performing areas.
The Inner Loop: Skill Learning
Once the fixture is placed, the robot enters the inner loop to learn the actual manipulation task. Using a variant of the Asynchronous Advantage Actor-Critic (A3C) algorithm, the robot processes RGB images and utilizes operational space control to master the skill within the constrained environment.
Physical Curriculum and Virtual Scaffolding
To ensure the robot does not become overly dependent on the physical fixture, the framework includes a curriculum-based "fading" process. The physical fixture is gradually replaced by a virtual potential field—a software-based "soft" constraint. This field is then slowly moved away, forcing the robot to transition from physical guidance to pure visual-motor coordination.
Experimental Evaluation
The system was evaluated in both the PyBullet simulation environment and on real-world hardware, including Franka Panda and Kinova Gen3 arms. The researchers focused on three challenging tasks: round peg insertion, wrench manipulation of a bolt, and shallow-depth insertion of a battery-like object into a mobile phone slot. Domain randomization was employed during simulation to ensure the models could generalize to the physical world despite the "reality gap."
Results and Discussion
The results demonstrate that scaffolding provides a dramatic speedup in learning. In peg insertion tasks, robots using an optimal fixture achieved a 20% success rate in just 5,000 steps, whereas those without a fixture required 40,000 steps to reach the same milestone.
Beyond speed, the scaffolding approach also raised the performance ceiling. Robots managed nearly 80% success on tasks where they previously plateaued at 20% without environmental assistance. Crucially, the system proved robust; models trained entirely in simulation with fixtures achieved 100% success in real-world transfer for insertion and wrench tasks. The virtual potential field curriculum was essential for this, allowing the robot to maintain a 70% success rate even after all physical help was removed.
Conclusion and Future Work
This work highlights the power of environmental modification in robotic learning. By allowing robots to "help themselves" through physical scaffolding, complex contact-rich tasks become significantly more accessible.
Future research directions include moving beyond predefined fixture geometries to allow robots to design or select their own scaffolds. Additionally, automating the "fading" speed of the curriculum and exploring the use of multiple or dynamic fixtures could further enhance the autonomy and versatility of the system in complex assembly environments.
Summary updated