Real-Time Stylized Humanoid Behavior Control through Interaction and Synchronization

Zhiyan Cao, Tianxu Bao, Zeyu Ren, Yunxin Fan, Ken Deng, Wenchuan Jia

TL;DR

This paper introduces a novel real-time teleoperation framework that allows a human operator to control a humanoid robot, enabling the robot to mimic the operator's motion style and achieve tasks while maintaining balance. The system leverages human perception and style analysis to create more natural and adaptable humanoid behaviors, bridging the gap between human intuition and robotic execution.

What Problem Does This Paper Address?

Simulating human-like perception and motion control in characters and robots is a long-standing challenge due to the inherent diversity and complexity of human behavior. Existing methods often struggle with real-time adaptability, robustness to human style variations, and maintaining balance, especially when dealing with differences in physical characteristics between humans and robots. This paper aims to address this by enabling a humanoid character to dynamically synchronize its motion with a human operator's style and intentions in real-time, making the control more intuitive and the resulting behavior more natural and robust.

Methods

The core of the proposed system is a style-based teleoperation framework that integrates human perception and analysis. Key components include:

  • Kinect Motion Capture and Data Processing: Real-time capture of human motion using Azure Kinect Body Tracking SDK, including joint angles and Center of Mass (COM) estimation. This data is then smoothed and interpolated to account for differences in capture and control loop frequencies.
  • Human Style Analysis: A systematic analysis of human behaviors is performed, categorizing styles into "task motions" and "balancing styles." This analysis identifies key features like ankle twists, leg motion planes, and relative positions of joints that define an individual's style.
  • Motion Optimization: This module takes the smoothed human motion data and optimizes it for the character. It extracts high-level style features and uses a style-based pose optimization to create a feasible, stylized, and balanced pose. This involves a lower-body retargeting law that accounts for differences in human and character body proportions.
  • Body Balance and Synchronization Controller: This component ensures the character maintains balance and synchronizes its motion with the operator. It employs a PID joint controller with gravity compensation and a novel COM and Root Virtual Controller (CR-VC) strategy. The CR-VC uses model-based torque compensation, drawing from simplified models like the Inverted Pendulum Model (IPM) and Linear Inverted Pendulum Model (LIPM), to adjust torques and achieve synchronized, balanced motion. This compensation is crucial for handling differences in physical scales between the human and the character.

Datasets

The paper does not explicitly mention training on specific datasets. Instead, it relies on real-time motion capture data from a human operator using the Azure Kinect Body Tracking SDK for both input and evaluation. The system is designed to work with live human motion data, and its performance is demonstrated through various experimental scenarios.

Key Insights and Ideas

  • Human Perception as a Control Input: The framework leverages the human operator's understanding of tasks and the environment to guide the character's actions, moving beyond purely kinematic or dynamic control.
  • Style Mimicking is Achievable in Real-Time: By analyzing and extracting high-level style features, the system can effectively replicate the operator's unique motion characteristics, leading to more personalized character animations.
  • CR-VC for Robust Synchronization: The proposed CR-VC strategy, with its model-based torque compensation, is critical for overcoming the challenges posed by differing physical scales between humans and robots, ensuring stable and synchronized teleoperation.
  • Generalization Across Tasks and Styles: The system demonstrates remarkable flexibility, adapting to various tasks (walking, kicking, squatting, jumping) and different motion styles (normal, sneaky, swagger) with different operators.
  • Balance Recovery Through Operator Intuition: The framework allows operators to intuitively guide the character through balance recovery maneuvers, showcasing a more natural human-in-the-loop control paradigm.

Key Results

  • The system successfully synchronizes human and character motion in real-time, with a maximum delay of 0.172 seconds in stepping situations when using torque compensation, which is within acceptable limits for human visual processing.
  • Experiments demonstrated the ability to mimic various walking styles (normal, sneaky, swagger) with high fidelity.
  • The system successfully enabled the character to perform target-based tasks like kicking a football, maintaining balance throughout.
  • The framework showed generalization capabilities across different operators with varying sizes and shapes, and for tasks that are not easily simplified by the underlying IPM/LIPM models.
  • Qualitative results show visually convincing and stylized humanoid behaviors that closely resemble the operator's movements.

Limitations and Future Work

  • Operator Motion Limitations: The model-based torque compensation (Section 6.3) restricts operator motions to those that fit simplified models like IPM and LIPM, limiting highly dynamic or abnormal movements.
  • Motion Feasibility Guarantee: The system does not fully guarantee the motion feasibility of the character, as it partially relies on the human operator's perception and understanding of tasks.
  • Future Work:
    • Develop strategies for solving high-dynamic motions with high computational efficiency.
  • Integrate a pattern and style database to predict next poses based on required tasks.
  • Explore multi-person situations to create more socialized and comprehensive strategies.
  • Enhance the system to better handle a wider range of operator motions and ensure motion feasibility.

Summary updated

BibTeX
@inproceedings{cao2022real,
  author = {Zhiyan Cao and Tianxu Bao and Zeyu Ren and Yunxin Fan and Ken Deng and Wenchuan Jia},
  title = {{Real-Time Stylized Humanoid Behavior Control through Interaction and Synchronization}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=y8PTF2AkYm},
}

Loading related content...

Figure