This paper, presented at ICRA 2022, tackles one of the "holy grails" of robotics: training deep reinforcement learning (DRL) agents directly on physical hardware in a reasonable amount of time. The authors demonstrate a system that learns to significantly improve the trajectory tracking of a 7-DOF Baxter robot arm in under two hours of real-world wall clock time.
TL;DR
The authors propose a method to improve robot trajectory tracking by learning a "reference correction" policy directly on a real 7-DOF manipulator in less than two hours. By using an "informed initialization" via a coarse learned dynamics model and bounding RL actions for safety, they achieve sub-centimeter end-effector accuracy and robust generalization to unseen payloads.
What Problem Does This Paper Address?
Robotic manipulators with flexible joints (like the Baxter robot) are increasingly popular because they are safer for human-robot collaboration. However, these joints introduce complex, non-linear dynamics that make precise control—specifically "trajectory tracking"—notoriously difficult.
While classical controllers (like PID or inverse dynamics) are the industry standard, they often struggle with these non-linearities or require exhaustive manual tuning. Deep Reinforcement Learning (DRL) offers a way to learn these complexities, but it usually requires millions of samples in simulation. Unfortunately, "sim-to-real" transfer often fails because creating a perfectly accurate simulation of a flexible-joint robot is nearly impossible. This paper addresses the need for a safe, sample-efficient way to learn control policies directly on the physical robot.
Methods
The core idea is to treat the DRL agent as an outer-loop controller. Instead of the agent outputting raw torques (which is dangerous and hard to learn), it outputs a small "correction" ($a$) that is added to the desired reference trajectory before it reaches the robot's internal (vendor-provided) controller.
Key technical components include:
- Soft Actor-Critic (SAC): An off-policy RL algorithm chosen for its sample efficiency and stability.
- Reference Correction: The policy modifies the input to the low-level controller. This keeps the method "controller-agnostic" and ensures the robot stays within its safe operating envelope.
- Beta Policy: Unlike standard SAC which uses a Gaussian distribution for actions, the authors use a Beta distribution. This naturally bounds actions between [0, 1], preventing the "squashing" issues often seen with Tanh-Gaussian policies.
- Informed Initialization: To avoid starting from scratch on the real robot, they record 20 minutes of random movement to train a "coarse" neural network dynamics model. They then pre-train the RL agent in this "learned simulation" before moving to the real hardware.
- Safety Measures: Actions are strictly bounded ($a_{max}$) and passed through a 4Hz low-pass filter to prevent jerky, hardware-damaging movements.
Datasets
The study does not use standard vision datasets but relies on real-time sensor data from a 7-DOF Baxter robot arm:
- Pre-training data: 200 random trajectories (approx. 20 minutes) used to build the coarse dynamics model.
- Real-world training: 1000 random trajectories executed over ~100 minutes.
- Evaluation: 100 unseen test trajectories, tested both with no load and with a 0.9 kg payload (which the agent never saw during training) to test generalization.
Key Insights and Ideas
- The Power of "Good Enough" Simulation: You don't need a high-fidelity physics engine like MuJoCo or Isaac Gym to pre-train. A simple neural network that learns "coarse" dynamics from just 20 minutes of real data is enough to give the RL agent a massive head start.
- Reference Correction vs. Direct Control: By "nudging" the reference rather than commanding torques, the system inherits the safety and stability of the underlying classical controller while the RL agent handles the "fine-tuning" of the non-linearities.
- L1 Norm for Rewards: The authors found that using an L1 norm (absolute error) for the reward function was superior to L2 (squared error). L2 caused the agent to "abuse" joints with smaller errors to compensate for larger ones, leading to unintuitive and unstable motions.
- Robot Persistence: The paper emphasizes "task-persistence"—the ability of the robot to collect its own data with minimal human intervention—as a requirement for practical real-world RL.
Key Results
The proposed Stochastic Reference Correction Policy (SRCP) outperformed the vendor-provided baseline significantly:
- Tracking Accuracy: Improved joint tracking accuracy by more than 3x.
- End-Effector Precision: Reduced the average end-effector tracking error from 3.12 cm (baseline) to just 0.66 cm.
- Generalization: When a 0.9 kg payload was added, the baseline error jumped to 4.35 cm, while the SRCP maintained a much tighter 1.19 cm error, despite never being trained with a payload.
- Learning Speed: With "Informed Initialization," the agent became twice as accurate as the baseline in just 18 minutes of real-world training.
Limitations and Future Work
- Latency: The low-pass filter used for safety introduces a small delay. While the RL agent learns to compensate for this, it is a trade-off between smoothness and instantaneous response.
- Hyperparameter Sensitivity: The maximum correction magnitude ($a_{max}$) is currently a fixed hyperparameter. If set too low, the agent can't correct enough; if too high, it might become unsafe.
- Future Directions: The authors suggest learning $a_{max}$ adaptively alongside the policy and incorporating adaptive safety constraints that monitor tracking error in real-time to further protect the hardware during the early stages of learning.
Summary updated