Introduction
The spherical pendulum is a classic benchmark in control theory, representing a "boss level" challenge due to its inherent instability and three-dimensional movement. In this work, researchers address the task of balancing and tracking 3D trajectories using an underactuated robotic arm. The setup is particularly difficult because the robot can only move the pendulum's base, not its joints directly, and must infer the system's full state from partial observations.
Problem Formulation
The authors identify three primary hurdles to mastering this task:
- Partial Observability: The controller only receives the pendulum's position, requiring it to reconstruct velocity information from historical data.
- Complex Trajectory Tracking: The robot is tasked with following intricate "eight-shaped" 3D paths rather than simply maintaining an upright position.
- The Cold Start Problem: Standard Reinforcement Learning (RL) often fails because the pendulum falls before the agent can collect meaningful rewards.
Methodology
To overcome these challenges, the team developed an automated curriculum learning framework based on CURROT (Curriculum Reinforcement Learning via Constrained Optimal Transport). This approach treats the curriculum as a dynamic distribution of tasks that evolves toward a target goal.
Two key technical innovations were introduced to enhance CURROT:
- Affine Metrics ($d_\Psi$): Instead of standard Euclidean distance, the authors utilized a Mahalanobis distance based on the system's Linear Time-Invariant (LTI) dynamics. This physics-aware metric ensures that task similarity reflects actual control difficulty.
- Directional Sampling: To combat the curse of dimensionality in high-dimensional task spaces (up to 399D), a new sampling scheme was implemented to efficiently find "descent directions" toward the target distribution.
The RL backbone utilizes Proximal Policy Optimization (PPO) trained within IsaacSim. To ensure successful sim-to-real transfer, the simulation meticulously models the Barrett WAM arm's tendon drives, friction, and communication latencies.
Key Insights and Results
A critical finding was that the choice of metric is paramount; a physics-aware distance allows the curriculum to progress through qualitatively relevant intermediate tasks. Furthermore, while advanced policy architectures can sometimes compensate for poor curricula, a robust curriculum (CURROT$_{AO}$) is essential for convergence in high-dimensional spaces.
The proposed method demonstrated significant improvements over baselines:
- Efficiency: Curriculum-based agents achieved task completion much faster than standard PPO.
- Sim-to-Real Success: On the physical Barrett WAM robot, the policy achieved an average tracking error of 2.28 cm, outperforming the PPO baseline (3.07 cm) and matching classical optimal control methods like Differential Dynamic Programming (DDP).
- Reliability: Unlike the baseline, which failed in some real-world trials, the CURROT$_{AO}$ seeds consistently succeeded.
Conclusion
This framework proves that automated curriculum learning can enable robots to master highly unstable, underactuated tasks. While a ~20% performance drop remains between simulation and reality, the method provides a robust path forward for complex dynamic control. Future work will explore even more dynamic scenarios, such as non-rigidly attached pendulums involving complex impact dynamics.
Summary updated