Learning Controller Gains on Bipedal Walking Robots via User Preferences

Noel Csomay-Shanklin, Maegan Tucker, Min Dai, Jenna Reher, Aaron D. Ames

TL;DR

This paper introduces a preference-based learning framework to automatically tune the complex controller gains for bipedal robots, significantly reducing the need for expert knowledge and manual tuning. By leveraging human user preferences, the method successfully enables stable and robust walking on both a planar and a 3D bipedal robot, demonstrating its effectiveness in high-dimensional action spaces.

What Problem Does This Paper Address?

Achieving stable and robust locomotion in bipedal robots is notoriously difficult, largely due to the extreme sensitivity of their behavior to controller gains. Traditionally, tuning these gains requires deep domain expertise and extensive trial-and-error, a process that is both time-consuming and non-intuitive, especially for complex controllers like Control Lyapunov Function Quadratic Programs (CLF-QPs). This paper addresses the challenge of efficiently and effectively finding optimal controller gains for bipedal robots, particularly for underactuated systems where the relationship between parameters and resulting behavior is complex and hard to predict. The goal is to democratize the process of tuning these controllers, making advanced robotic locomotion more accessible.

Methods

The core of this work is a preference-based learning framework that learns an underlying utility function representing human satisfaction with a robot's walking behavior. Instead of relying on a predefined reward function, the system queries a human operator for pairwise preferences (e.g., "Do you prefer gait A over gait B?") and ordinal labels (e.g., "Rate this gait as very bad, neutral, or very good").

This feedback is used to approximate a posterior distribution over the utility function. The framework then employs Thompson sampling to intelligently select the next set of controller gains to test, aiming to minimize regret (i.e., find the optimal gains quickly).

The paper applies this framework to two specific controllers:

  • CLF-QP+: Used on the AMBER robot, this controller uses a quadratic program to enforce stability conditions derived from Control Lyapunov Functions.
  • ID-CLF-QP+: A more complex, inverse dynamics version used on the Cassie robot, which accounts for more degrees of freedom and challenges in underactuated locomotion.

The controller gains are parameterized as an "action" in a multi-dimensional space, and the learning algorithm searches this space for actions that lead to preferred walking behaviors.

Datasets

The paper does not introduce new datasets. Instead, it uses experimental data generated from real-world robot trials on two platforms:

  • AMBER robot: A 5 DOF planar biped.
  • Cassie robot: A 22 DOF 3D biped.

The "dataset" in this context is the collection of user preferences and ordinal labels gathered during the experiments, which are used to train the preference learning model.

Key Insights and Ideas

  • Human preferences are a powerful, low-dimensional signal for high-dimensional control problems: The paper demonstrates that subjective human feedback, even when noisy, is sufficient to guide the learning process towards stable and robust robotic locomotion, bypassing the need for explicit, objective reward functions.
  • Ordinal labels significantly improve sample efficiency: Supplementing pairwise preferences with ordinal labels (very bad, neutral, very good) helps the learning algorithm converge faster, especially in complex, high-dimensional action spaces.
  • Preference-based learning is application-agnostic: The framework's success on two distinct robots (AMBER and Cassie) with two different controllers (CLF-QP+ and ID-CLF-QP+) highlights its generality and applicability to various bipedal locomotion control problems.
  • Bridging the gap between theoretical controllers and practical realization: Controllers like CLF-QPs offer theoretical guarantees but are challenging to tune in practice. This work shows how preference learning can make these powerful controllers usable on physical hardware.

Key Results

  • Stable walking achieved in hours: For both AMBER and Cassie, the robots transitioned from stumbling or inability to walk unassisted to stable, robust locomotion within a matter of hours (50 iterations for AMBER, 100 for Cassie).
  • Learned gains matched user preferences: The "best" actions identified by the learning algorithm consistently corresponded to the walking behaviors that the human operators preferred the most.
  • Improved tracking and reduced torque chatter: The learned optimal gains resulted in significantly better tracking of desired robot trajectories and a substantial reduction in undesirable torque chatter compared to initial, poorly performing gains.
  • Demonstrated robustness: The learned gaits were robust to external disturbances, enabling unassisted walking.

Limitations and Future Work

  • Predefined action space bounds: The current formulation requires the action space (i.e., the range of controller gains) to be predefined with set bounds. This is challenging because the relationship between gains and behavior is often unpredictable.
  • Discretized action space: The learning algorithm searches a discretized action space. Future work could involve adapting the granularity of this space based on uncertainty.

Future work directions include:

  • Dynamic action space shifting: Modifying the framework to adjust the action space boundaries based on user preferences.
  • Adapting granularity: Dynamically adjusting the discretization of the action space.
  • Extending to other platforms: Applying the method to different robotic systems.
  • Hybrid approaches: Combining preference-based learning with metric-based optimization techniques.
  • Multi-layered tuning: Addressing more complex parameter tuning tasks.

Summary updated

Abstract

Experimental demonstration of complex robotic behaviors relies heavily on finding the correct controller gains. This painstaking process is often completed by a domain expert, requiring deep knowledge of the relationship between parameter values and the resulting behavior of the system. Even when such knowledge is possessed, it can take significant effort to navigate the nonintuitive landscape of possible parameter combinations. In this work, we explore the extent to which preference-based learning can be used to optimize controller gains online by repeatedly querying the user for their preferences. This general methodology is applied to two variants of control Lyapunov function based nonlinear controllers framed as quadratic programs, which provide theoretical guarantees but are challenging to realize in practice. These controllers are successfully demonstrated both on the planar underactuated biped, AMBER, and on the 3D underactuated biped, Cassie. We experimentally evaluate the performance of the learned controllers and show that the proposed method is repeatably able to learn gains that yield stable and robust locomotion.
BibTeX
@inproceedings{csomayshanklin2022learning,
  author = {Noel Csomay-Shanklin and Maegan Tucker and Min Dai and Jenna Reher and Aaron D. Ames},
  title = {{Learning Controller Gains on Bipedal Walking Robots via User Preferences}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  doi = {10.1109/icra46639.2022.9811541},
}
Figure