Tracking Control for a Spherical Pendulum via Curriculum Reinforcement Learning

Pascal Klink, Florian Wolf, Kai Ploeger, Jan Peters, Joni PajarinenarXiv 2023 · DOI · arXiv

Introduction

The spherical pendulum is a classic benchmark in control theory, representing a "boss level" challenge due to its inherent instability and three-dimensional movement. In this work, researchers address the task of balancing and tracking 3D trajectories using an underactuated robotic arm. The setup is particularly difficult because the robot can only move the pendulum's base, not its joints directly, and must infer the system's full state from partial observations.

Comparison of the spherical pendulum setup in the IsaacSim simulation environment (left) and the physical Barrett WAM deployment (right).

Problem Formulation

The authors identify three primary hurdles to mastering this task:

  1. Partial Observability: The controller only receives the pendulum's position, requiring it to reconstruct velocity information from historical data.
  2. Complex Trajectory Tracking: The robot is tasked with following intricate "eight-shaped" 3D paths rather than simply maintaining an upright position.
  3. The Cold Start Problem: Standard Reinforcement Learning (RL) often fails because the pendulum falls before the agent can collect meaningful rewards.

Methodology

To overcome these challenges, the team developed an automated curriculum learning framework based on CURROT (Curriculum Reinforcement Learning via Constrained Optimal Transport). This approach treats the curriculum as a dynamic distribution of tasks that evolves toward a target goal.

Reinforcement learning framework for trajectory tracking, mapping observations and target trajectories to robot joint actions.

Two key technical innovations were introduced to enhance CURROT:

  • Affine Metrics ($d_\Psi$): Instead of standard Euclidean distance, the authors utilized a Mahalanobis distance based on the system's Linear Time-Invariant (LTI) dynamics. This physics-aware metric ensures that task similarity reflects actual control difficulty.
  • Directional Sampling: To combat the curse of dimensionality in high-dimensional task spaces (up to 399D), a new sampling scheme was implemented to efficiently find "descent directions" toward the target distribution.

The RL backbone utilizes Proximal Policy Optimization (PPO) trained within IsaacSim. To ensure successful sim-to-real transfer, the simulation meticulously models the Barrett WAM arm's tendon drives, friction, and communication latencies.

Key Insights and Results

A critical finding was that the choice of metric is paramount; a physics-aware distance allows the curriculum to progress through qualitatively relevant intermediate tasks. Furthermore, while advanced policy architectures can sometimes compensate for poor curricula, a robust curriculum (CURROT$_{AO}$) is essential for convergence in high-dimensional spaces.

The proposed method demonstrated significant improvements over baselines:

  • Efficiency: Curriculum-based agents achieved task completion much faster than standard PPO.
  • Sim-to-Real Success: On the physical Barrett WAM robot, the policy achieved an average tracking error of 2.28 cm, outperforming the PPO baseline (3.07 cm) and matching classical optimal control methods like Differential Dynamic Programming (DDP).
  • Reliability: Unlike the baseline, which failed in some real-world trials, the CURROT$_{AO}$ seeds consistently succeeded.

Conclusion

This framework proves that automated curriculum learning can enable robots to master highly unstable, underactuated tasks. While a ~20% performance drop remains between simulation and reality, the method provides a robust path forward for complex dynamic control. Future work will explore even more dynamic scenarios, such as non-rigidly attached pendulums involving complex impact dynamics.

Summary updated

Abstract

Reinforcement Learning (RL) allows learning non-trivial robot control laws purely from data. However, many successful applications of RL have relied on ad-hoc regularizations, such as hand-crafted curricula, to regularize the learning performance. In this paper, we pair a recent algorithm for automatically building curricula with RL on massively parallelized simulations to learn a tracking controller for a spherical pendulum on a robotic arm via RL. Through an improved optimization scheme that better respects the non-Euclidean task structure, we allow the method to reliably generate curricula of trajectories to be tracked, resulting in faster and more robust learning compared to an RL baseline that does not exploit this form of structured learning. The learned policy matches the performance of an optimal control baseline on the real system, demonstrating the potential of curriculum RL to jointly learn state estimation and control for non-linear tracking tasks.
BibTeX
@misc{klink2023tracking,
  author = {Pascal Klink and Florian Wolf and Kai Ploeger and Jan Peters and Joni Pajarinen},
  title = {{Tracking Control for a Spherical Pendulum via Curriculum Reinforcement Learning}},
  year = {2023},
  eprint = {2309.14096},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.14096},
}
Figure