Real-Robot Deep Reinforcement Learning: Improving Trajectory Tracking of Flexible-Joint Manipulator with Reference Correction

Dmytro Pavlichenko, Sven Behnke

Abstract: Flexible-joint manipulators are governed by complex nonlinear dynamics, defining a challenging control problem. In this work, we propose an approach to learn an outer-loop joint trajectory tracking controller with deep reinforcement learning. The controller represented by a stochastic policy is learned in under two hours directly on the real robot. This is achieved through bounded reference correction actions and use of a model-free off-policy learning method. In addition, an informed policy initialization is proposed, where the agent is pre-trained in a learned simulation. We test our approach on the 7 DOF manipulator of a Baxter robot. We demonstrate that the proposed method is capable of consistent learning across multiple runs when applied directly on the real robot. Our method yields a policy which significantly improves the trajectory tracking accuracy in comparison to the vendor-provided controller, generalizing to an unseen payload.

Introduction

Robotic manipulators with flexible joints, such as the Baxter robot, offer safety benefits for human-robot interaction but present significant control challenges due to non-linear dynamics. While classical controllers like PID are standard, they often struggle with these complexities. Deep Reinforcement Learning (DRL) can model such dynamics but typically requires millions of samples in simulation, and "sim-to-real" transfer is often unreliable. This paper presents a safe, sample-efficient method to learn control policies directly on physical hardware in less than two hours.

Methodology

The core of the approach is an outer-loop controller that performs "reference correction." Rather than outputting raw torques, the DRL agent—a stochastic policy trained using Soft Actor-Critic (SAC)—outputs a small correction value ($a$) added to the desired reference trajectory ($q_r$). This resulting modified trajectory ($q_f$) is then passed to the robot's internal classical controller.

The control loop architecture, showing the outer-loop stochastic policy providing corrections to the inner-loop classical controller.

To ensure safety and efficiency, the authors implemented:

  1. Informed Initialization: A coarse dynamics model is trained on 20 minutes of random movement data to pre-train the agent before real-world deployment.
  2. Beta Policy: Using a Beta distribution for actions naturally bounds them, avoiding issues with traditional Gaussian policies.
  3. Safety Constraints: Actions are strictly bounded and low-pass filtered to prevent hardware damage.
Conceptual illustration of reference correction: the policy adjusts the reference to bring the resulting execution closer to the target.

Experimental Results

The proposed Stochastic Reference Correction Policy (SRCP) was evaluated on a 7-DOF Baxter arm. Key findings include:

  • Accuracy: Joint tracking accuracy improved by over 3x. End-effector tracking error was reduced from 3.12 cm (baseline) to 0.66 cm.
  • Generalization: The policy maintained high precision (1.19 cm error) when a 0.9 kg payload was added, whereas the baseline error increased to 4.35 cm.
  • Efficiency: With informed initialization, the agent surpassed baseline performance in just 18 minutes of real-world training.

Conclusion

This research demonstrates that DRL can be effectively and safely applied directly to real-world robotic tasks by framing the agent as an augmenting outer-loop controller. The use of learned "coarse" simulations for initialization provides a practical path forward for rapid real-robot learning without the need for high-fidelity physics engines.

Summary updated

Abstract

Flexible-joint manipulators are governed by complex nonlinear dynamics, defining a challenging control problem. In this work, we propose an approach to learn an outer-loop joint trajectory tracking controller with deep reinforcement learning. The controller represented by a stochastic policy is learned in under two hours directly on the real robot. This is achieved through bounded reference correction actions and use of a model-free off-policy learning method. In addition, an informed policy initialization is proposed, where the agent is pre-trained in a learned simulation. We test our approach on the 7 DOF manipulator of a Baxter robot. We demonstrate that the proposed method is capable of consistent learning across multiple runs when applied directly on the real robot. Our method yields a policy which significantly improves the trajectory tracking accuracy in comparison to the vendor-provided controller, generalizing to an unseen payload.
BibTeX
@inproceedings{pavlichenko2022real,
  author = {Dmytro Pavlichenko and Sven Behnke},
  title = {{Real-Robot Deep Reinforcement Learning: Improving Trajectory Tracking of Flexible-Joint Manipulator with Reference Correction}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=esxphkNqhixA},
}
Figure