Abstract: Flexible-joint manipulators are governed by complex nonlinear dynamics, defining a challenging control problem. In this work, we propose an approach to learn an outer-loop joint trajectory tracking controller with deep reinforcement learning. The controller represented by a stochastic policy is learned in under two hours directly on the real robot. This is achieved through bounded reference correction actions and use of a model-free off-policy learning method. In addition, an informed policy initialization is proposed, where the agent is pre-trained in a learned simulation. We test our approach on the 7 DOF manipulator of a Baxter robot. We demonstrate that the proposed method is capable of consistent learning across multiple runs when applied directly on the real robot. Our method yields a policy which significantly improves the trajectory tracking accuracy in comparison to the vendor-provided controller, generalizing to an unseen payload.
Introduction
Robotic manipulators with flexible joints, such as the Baxter robot, offer safety benefits for human-robot interaction but present significant control challenges due to non-linear dynamics. While classical controllers like PID are standard, they often struggle with these complexities. Deep Reinforcement Learning (DRL) can model such dynamics but typically requires millions of samples in simulation, and "sim-to-real" transfer is often unreliable. This paper presents a safe, sample-efficient method to learn control policies directly on physical hardware in less than two hours.
Methodology
The core of the approach is an outer-loop controller that performs "reference correction." Rather than outputting raw torques, the DRL agent—a stochastic policy trained using Soft Actor-Critic (SAC)—outputs a small correction value ($a$) added to the desired reference trajectory ($q_r$). This resulting modified trajectory ($q_f$) is then passed to the robot's internal classical controller.
To ensure safety and efficiency, the authors implemented:
- Informed Initialization: A coarse dynamics model is trained on 20 minutes of random movement data to pre-train the agent before real-world deployment.
- Beta Policy: Using a Beta distribution for actions naturally bounds them, avoiding issues with traditional Gaussian policies.
- Safety Constraints: Actions are strictly bounded and low-pass filtered to prevent hardware damage.
Experimental Results
The proposed Stochastic Reference Correction Policy (SRCP) was evaluated on a 7-DOF Baxter arm. Key findings include:
- Accuracy: Joint tracking accuracy improved by over 3x. End-effector tracking error was reduced from 3.12 cm (baseline) to 0.66 cm.
- Generalization: The policy maintained high precision (1.19 cm error) when a 0.9 kg payload was added, whereas the baseline error increased to 4.35 cm.
- Efficiency: With informed initialization, the agent surpassed baseline performance in just 18 minutes of real-world training.
Conclusion
This research demonstrates that DRL can be effectively and safely applied directly to real-world robotic tasks by framing the agent as an augmenting outer-loop controller. The use of learned "coarse" simulations for initialization provides a practical path forward for rapid real-robot learning without the need for high-fidelity physics engines.
Summary updated