TL;DR
This paper introduces a preference-based learning framework to automatically tune the complex controller gains for bipedal robots, significantly reducing the need for expert knowledge and manual tuning. By leveraging human user preferences, the method successfully enables stable and robust walking on both a planar and a 3D bipedal robot, demonstrating its effectiveness in high-dimensional action spaces.
What Problem Does This Paper Address?
Achieving stable and robust locomotion in bipedal robots is notoriously difficult, largely due to the extreme sensitivity of their behavior to controller gains. Traditionally, tuning these gains requires deep domain expertise and extensive trial-and-error, a process that is both time-consuming and non-intuitive, especially for complex controllers like Control Lyapunov Function Quadratic Programs (CLF-QPs). This paper addresses the challenge of efficiently and effectively finding optimal controller gains for bipedal robots, particularly for underactuated systems where the relationship between parameters and resulting behavior is complex and hard to predict. The goal is to democratize the process of tuning these controllers, making advanced robotic locomotion more accessible.
Methods
The core of this work is a preference-based learning framework that learns an underlying utility function representing human satisfaction with a robot's walking behavior. Instead of relying on a predefined reward function, the system queries a human operator for pairwise preferences (e.g., "Do you prefer gait A over gait B?") and ordinal labels (e.g., "Rate this gait as very bad, neutral, or very good").
This feedback is used to approximate a posterior distribution over the utility function. The framework then employs Thompson sampling to intelligently select the next set of controller gains to test, aiming to minimize regret (i.e., find the optimal gains quickly).
The paper applies this framework to two specific controllers:
- CLF-QP+: Used on the AMBER robot, this controller uses a quadratic program to enforce stability conditions derived from Control Lyapunov Functions.
- ID-CLF-QP+: A more complex, inverse dynamics version used on the Cassie robot, which accounts for more degrees of freedom and challenges in underactuated locomotion.
The controller gains are parameterized as an "action" in a multi-dimensional space, and the learning algorithm searches this space for actions that lead to preferred walking behaviors.
Datasets
The paper does not introduce new datasets. Instead, it uses experimental data generated from real-world robot trials on two platforms:
- AMBER robot: A 5 DOF planar biped.
- Cassie robot: A 22 DOF 3D biped.
The "dataset" in this context is the collection of user preferences and ordinal labels gathered during the experiments, which are used to train the preference learning model.
Key Insights and Ideas
- Human preferences are a powerful, low-dimensional signal for high-dimensional control problems: The paper demonstrates that subjective human feedback, even when noisy, is sufficient to guide the learning process towards stable and robust robotic locomotion, bypassing the need for explicit, objective reward functions.
- Ordinal labels significantly improve sample efficiency: Supplementing pairwise preferences with ordinal labels (very bad, neutral, very good) helps the learning algorithm converge faster, especially in complex, high-dimensional action spaces.
- Preference-based learning is application-agnostic: The framework's success on two distinct robots (AMBER and Cassie) with two different controllers (CLF-QP+ and ID-CLF-QP+) highlights its generality and applicability to various bipedal locomotion control problems.
- Bridging the gap between theoretical controllers and practical realization: Controllers like CLF-QPs offer theoretical guarantees but are challenging to tune in practice. This work shows how preference learning can make these powerful controllers usable on physical hardware.
Key Results
- Stable walking achieved in hours: For both AMBER and Cassie, the robots transitioned from stumbling or inability to walk unassisted to stable, robust locomotion within a matter of hours (50 iterations for AMBER, 100 for Cassie).
- Learned gains matched user preferences: The "best" actions identified by the learning algorithm consistently corresponded to the walking behaviors that the human operators preferred the most.
- Improved tracking and reduced torque chatter: The learned optimal gains resulted in significantly better tracking of desired robot trajectories and a substantial reduction in undesirable torque chatter compared to initial, poorly performing gains.
- Demonstrated robustness: The learned gaits were robust to external disturbances, enabling unassisted walking.
Limitations and Future Work
- Predefined action space bounds: The current formulation requires the action space (i.e., the range of controller gains) to be predefined with set bounds. This is challenging because the relationship between gains and behavior is often unpredictable.
- Discretized action space: The learning algorithm searches a discretized action space. Future work could involve adapting the granularity of this space based on uncertainty.
Future work directions include:
- Dynamic action space shifting: Modifying the framework to adjust the action space boundaries based on user preferences.
- Adapting granularity: Dynamically adjusting the discretization of the action space.
- Extending to other platforms: Applying the method to different robotic systems.
- Hybrid approaches: Combining preference-based learning with metric-based optimization techniques.
- Multi-layered tuning: Addressing more complex parameter tuning tasks.
Summary updated