Self-organized Learning from Synthetic and Real-World Data for a Humanoid Exercise Robot

Nicolas Duczek, Matthias Kerzel, Philipp Allgeuer, Stefan WermterFrontiers Robotics AI 2022 · DOI · OpenReview

Introduction

Humanoid robots are increasingly viewed as viable candidates for health coaching and physical therapy. However, deploying these robots in real-world scenarios requires overcoming the "stability-plasticity" dilemma. A robot trainer must be able to learn complex temporal sequences of exercises, adapt to the diverse body shapes (morphologies) of different users, and avoid catastrophic forgetting—where learning a new user's movements erases the knowledge of previous trainees.

Humanoid robot Pepper providing exercise feedback to a user.

To address these challenges, researchers have developed Subnode-GWR, a novel neural architecture designed for humanoid robots like Pepper. This system allows for incremental learning and personalized adaptation, enabling the robot to act as an effective personal trainer.

Methodology: The Subnode-GWR Architecture

The proposed system builds upon the Grow-When-Required (GWR) network, which dynamically adds nodes to represent new data. The Subnode-GWR integrates several key innovations to handle the complexities of human motion.

Temporal Memory with Gamma-GWR

Standard GWR networks often struggle with the temporal nature of exercises. By integrating Gamma-GWR principles, the network utilizes recurrent connections. This provides the system with a "memory" of previous frames, allowing it to understand not just static poses, but also the velocity and direction of movement.

The Subnode Mechanism

The core innovation is the "subnode" mechanism. Traditional self-organizing maps can fail when a user's body shape doesn't match the training data. Instead of adding new nodes to a global map for every variation, the system creates subnodes attached to existing exercise states. When a new user interacts with the robot, it creates a personalized mapping (subnode) for that specific user’s limb proportions. This effectively decouples the logic of the exercise (the sequence of states) from the anatomy of the individual user.

Overview of the Subnode-GWR system architecture for real-time exercise monitoring.

Real-Time Feedback Loop

The system employs OpenPose to extract the user's skeleton in real-time. The robot compares the extracted pose and velocity against the "ideal" trajectory stored in the Subnode-GWR. If the user's performance deviates beyond a set threshold, the robot provides immediate feedback. This includes visual cues on its tablet (e.g., highlighting joints in red) and verbal instructions.

Experimental Evaluation

The system was validated using two distinct datasets:

  1. Virtual-Squat (Synthetic): Created using Blender and MakeHuman, this dataset includes 10 diverse virtual avatars with varying heights, weights, and genders. It contains 240 videos depicting both correct squats and five common errors, such as "arms too low" or "too fast."
  2. Office-Squat (Real-World): This dataset consists of 60 videos recorded in a standard office environment, testing the system's robustness against background noise and varied lighting conditions.

Key Results

The Subnode-GWR demonstrated significant improvements over baseline models:

  • Prediction Accuracy: The model achieved a joint-wise prediction error of only 1.73 pixels, outperforming standard Gamma-GWR (2.88 pixels).
  • Classification: It reached an average accuracy of 88% in classifying exercise correctness, maintaining performance even when users were rotated or moved within the camera's view.
  • Efficiency: Inference time for the Subnode-GWR is approximately 0.8ms per frame. Combined with OpenPose (~45ms), the system operates at interactive speeds on standard hardware.

Discussion and Future Work

A major strength of the Subnode-GWR is its ability to perform "one-shot" adaptation. Once a user performs a single correct baseline exercise, the subnodes are fixed, allowing for consistent coaching in future sessions without further training. Furthermore, the episodic memory structure prevents the network from getting "stuck" in loops during held poses.

While the current system requires an initial expert demonstration and manual tuning of error thresholds, future research aims to automate these processes. The researchers suggest this architecture could be applied beyond exercise to tasks like sign language recognition or complex gesture-based human-robot interaction.

Summary updated

Abstract

p>We propose a neural learning approach for a humanoid exercise robot that can automatically analyze and correct physical exercises. Such an exercise robot should be able to train many different human partners over time and thus requires the ability for lifelong learning. To this end, we develop a modified Grow-When-Required (GWR) network with recurrent connections, episodic memory and a novel subnode mechanism for learning spatiotemporal relationships of body movements and poses. Once an exercise is successfully demonstrated, the information of pose and movement per frame is stored in the Subnode-GWR network. For every frame, the current pose and motion pair is compared against a predicted output of the GWR, allowing for feedback not only on the pose but also on the velocity of the motion. Since both the pose and motion depend on a user’s body morphology, the exercise demonstration by one individual cannot easily be used as a reference for further users. We allow the GWR to grow online with each further demonstration. The subnode mechanism ensures that exercise information for individual humans is stored and retrieved correctly and is not forgotten over time. In the application scenario, a physical exercise is performed in the presence of an expert like a physiotherapist and then used as a reference for a humanoid robot like Pepper to give feedback on further executions of the same exercise. For evaluation, we developed a new synthetic exercise dataset with virtual avatars. We also test our method on real-world data recorded in an office scenario. Overall, we claim that our novel GWR-based architecture can use a learned exercise reference for different body variations through incremental online learning while preventing catastrophic forgetting, enabling an engaging long-term human-robot experience with a humanoid robot.</p>
BibTeX
@article{duczek2022self,
  author = {Nicolas Duczek and Matthias Kerzel and Philipp Allgeuer and Stefan Wermter},
  title = {{Self-organized Learning from Synthetic and Real-World Data for a Humanoid Exercise Robot}},
  journal = {Frontiers Robotics AI},
  year = {2022},
  doi = {10.3389/frobt.2022.669719},
}

Loading related content...

Figure