Robotic Arm Control and Task Training through Deep Reinforcement Learning

A Franceschetti, Elisa Tosello, Nicola Castaman, Stefano GhidoniarXiv 2020 · DOI · arXiv

Abstract

This paper proposes a detailed and extensive comparison of the Trust Region Policy Optimization (TRPO) and Deep Q-Network with Normalized Advantage Functions (DQN-NAF) with respect to other state-of-the-art algorithms, namely Deep Deterministic Policy Gradient (DDPG) and Vanilla Policy Gradient (VPG). Comparisons demonstrate that the former have better performances than the latter when asking robotic arms to accomplish manipulation tasks such as reaching a random target pose and pick-and-placing an object. Both simulated and real-world experiments are provided. Simulation lets us show the procedures that we adopted to precisely estimate the algorithms' hyperparameters and to correctly design good policies. Real-world experiments demonstrate that our policies, if correctly trained on simulation, can be transferred and executed in a real environment with almost no changes.

Introduction

Training robotic arms for flexible, autonomous tasks in dynamic environments remains a significant challenge in robotics. Traditional control methods often rely on rigid geometric models that fail when environmental parameters shift. Reinforcement Learning (RL) offers a promising alternative by allowing agents to learn through trial and error. However, the high Degrees of Freedom (DoF) in modern manipulators create high-dimensional state spaces that are computationally expensive. This research evaluates which Deep Reinforcement Learning (DRL) algorithms are most robust for these tasks and investigates the feasibility of bridging the "reality gap" between physics simulators and physical hardware.

Methodology

The study compares four primary DRL algorithms:

  1. Trust Region Policy Optimization (TRPO): A stochastic method using a "trust region" to maintain behavioral stability during updates.
  2. Deep Q-Network with Normalized Advantage Functions (DQN-NAF): A deterministic approach adapting Q-learning for continuous action spaces.
  3. Deep Deterministic Policy Gradient (DDPG): A baseline for continuous control.
  4. Vanilla Policy Gradient (VPG): A fundamental RL baseline.

Experiments were conducted using a UR5 manipulator (6 DoF) equipped with a Robotiq S Model gripper (4 DoF), totaling 10 DoF. Training data was actively generated within the MuJoCo physics engine. For the complex "Pick & Place" task, the researchers implemented reward shaping based on three geometric distances: end-effector to object, fingers to the object's center of mass, and object to the final goal.

Comparative Analysis of DRL Algorithms

The results indicate that algorithm choice is more critical than neural network architecture. While DQN-NAF excelled in simple reaching tasks—solving the environment in under 700 episodes with a final average return of -2.36—it struggled with more complex interactions.

Average return across episodes for the UR5Picker-v2 task, highlighting TRPO's stability and performance.

For the Pick & Place task, TRPO emerged as the clear winner with a final average return of 324.97. TRPO's stability and monotonic improvement theory allowed it to handle the stochastic nature of physical contacts more effectively than DDPG, which proved highly unstable and difficult to tune. TRPO even developed a sophisticated "tilting" strategy to lift objects more securely.

Sim-to-Real Transfer

A core contribution of this work is the successful transfer of simulated policies to physical hardware. By matching the simulation's control frequency (125Hz) to the real-world controller and incorporating proper joint damping, the researchers achieved a 100% success rate on the physical UR5 robot for both reaching and pick-and-place tasks.

The physical UR5 robotic arm successfully executing a pick-and-place task using a policy trained entirely in simulation.

Real-world object tracking was facilitated by a Microsoft Kinect sensor and AprilTag fiducial markers, allowing the robot to follow targets even while they were in motion.

Conclusion

This study confirms that TRPO and DQN-NAF provide robust frameworks for robotic manipulation. While TRPO is more versatile for multi-stage tasks involving physical contact, DQN-NAF offers rapid convergence for reaching tasks. The 100% success rate in real-world validation proves that high-fidelity simulations like MuJoCo can effectively train policies for immediate deployment on physical hardware. Future work will explore Hindsight Experience Replay (HER) for sparse rewards and real-world fine-tuning to achieve smoother, more human-like grasping motions.

Summary updated

Abstract

This paper proposes a detailed and extensive comparison of the Trust Region Policy Optimization and DeepQ-Network with Normalized Advantage Functions with respect to other state of the art algorithms, namely Deep Deterministic Policy Gradient and Vanilla Policy Gradient. Comparisons demonstrate that the former have better performances then the latter when asking robotic arms to accomplish manipulation tasks such as reaching a random target pose and pick &placing an object. Both simulated and real-world experiments are provided. Simulation lets us show the procedures that we adopted to precisely estimate the algorithms hyper-parameters and to correctly design good policies. Real-world experiments let show that our polices, if correctly trained on simulation, can be transferred and executed in a real environment with almost no changes.
BibTeX
@misc{franceschetti2020robotic,
  author = {A Franceschetti and Elisa Tosello and Nicola Castaman and Stefano Ghidoni},
  title = {{Robotic Arm Control and Task Training through Deep Reinforcement Learning}},
  year = {2020},
  eprint = {2005.02632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.02632},
}
Figure