Robust Agility via Learned Zero Dynamics Policies

Noel Csomay-Shanklin, William D. Compton, Ivan Dario Jimenez Rodriguez, Eric R. Ambrose, Yisong Yue, Aaron D. Ames

Abstract

Controlling underactuated robotic systems requires balancing computational efficiency with physical robustness. This paper introduces Zero Dynamics Policies (ZDP), a framework that integrates the formal stability of control theory with the representational power of machine learning. By learning a low-dimensional manifold that is invariant under optimal control actions, we enable a 3D hopping robot to execute agile maneuvers—such as climbing stairs and crossing narrow bridges—with high robustness and minimal online computational overhead.

Introduction

The control of underactuated systems, where the number of control inputs is less than the degrees of freedom, remains a fundamental challenge in robotics. Legged locomotion is a primary example, requiring precise coordination of actuated joints to influence unactuated global dynamics (e.g., the center of mass). Existing approaches typically fall into three categories: Model Predictive Control (MPC), which is computationally intensive; Reinforcement Learning (RL), which often lacks physical transparency; and heuristics, which may lack formal stability guarantees.

This work proposes a "middle ground" that exploits the physical structure of underactuation. By focusing on the zero dynamics of the system, we can achieve the performance of optimal control with the execution speed of simple heuristics.

Methodology: Zero Dynamics Policies

The core of the ZDP framework involves decomposing the robot's state into actuated coordinates ($\eta$) and unactuated coordinates ($z$). The objective is to learn a mapping, $\psi_\theta(z)$, that defines the target configuration for the actuated joints based on the current unactuated state.

The Zero Dynamics Manifold

We define a manifold $\mathcal{M}_\psi$ where the actuated states are constrained as a function of the unactuated ones. Stability of the entire system is guaranteed if two conditions are met:

  1. Invariance: If the robot starts on the manifold and takes an optimal control step, it must remain on the manifold.
  2. Stability: The "zero dynamics"—the motion of the system while restricted to the manifold—must be stable.
The ZDP framework requires the learned manifold to be both invariant under optimal control and internally stable.

Learning via Optimal Control

Rather than manual tuning, the manifold is learned by training a neural network to approximate trajectories generated by Iterative LQR (iLQR). The network is optimized to represent a manifold that is invariant under these optimal actions. This allows the controller to "remember" optimal behaviors, enabling 1kHz control loops on standard hardware without solving complex optimization problems in real-time.

Experimental Validation

The ZDP framework was validated using the ARCHER 3D hopping robot. Training data was generated synthetically by sampling the unactuated state space and solving iLQR problems. The resulting policy was evaluated through over 3,000 physical hops across diverse and challenging terrains.

Key Results

The robot demonstrated exceptional agility and robustness in several scenarios:

  • Disturbance Rejection: ARCHER maintained stability while hopping on a treadmill at 1 mph, even when subjected to external shoves.
  • Terrain Agility: The robot successfully climbed 1.5-inch stairs and descended 20° ramps without the aid of vision sensors.
  • Precision Maneuvering: The system exhibited sufficient accuracy to traverse a narrow 2x4 wooden bridge.
Experimental snapshots of the ARCHER robot performing (a) hopping, (b) stair climbing, (c) disturbance rejection, and (d) bridge crossing.

Discussion and Future Work

While ZDPs offer a robust and efficient control solution, some challenges remain. The training process is computationally expensive due to the need for repeated iLQR queries. Additionally, the solver currently requires a stabilizing initial guess (e.g., from a Raibert heuristic) to converge. Future research will explore integrating this structured approach with Reinforcement Learning to handle more complex environments and extending the framework to multi-legged platforms like quadrupeds.

Conclusion

Zero Dynamics Policies provide a powerful framework for achieving robust agility in underactuated robots. By embedding optimal control behavior into a learned, stable manifold, we achieve high-performance locomotion that is both computationally efficient and physically grounded.

Summary updated

BibTeX
@inproceedings{csomayshanklin2024robust,
  author = {Noel Csomay-Shanklin and William D. Compton and Ivan Dario Jimenez Rodriguez and Eric R. Ambrose and Yisong Yue and Aaron D. Ames},
  title = {{Robust Agility via Learned Zero Dynamics Policies}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801322},
}

Loading related content...

Figure