TL;DR
This paper introduces a novel method for estimating human arm pose using only a single smartwatch, achieving a 40% reduction in prediction error compared to prior work. By combining this accurate pose estimation with speech recognition, they create a ubiquitous and intuitive interface for robot control and teleoperation, enabling users to intervene in robot tasks or train new behaviors.
What Problem Does This Paper Address?
Controlling and teaching robots often requires intuitive interfaces to capture human motion. Traditional motion capture systems are expensive and stationary. While consumer-grade hardware like VR headsets offer alternatives, they can be cumbersome. Existing methods using single smartwatches for arm pose estimation have suffered from low accuracy. This paper aims to bridge this gap by developing a highly accurate, low-cost, and ubiquitous method for human arm pose estimation from a single smartwatch, paving the way for more accessible human-robot interaction.
Methods
The core of this work lies in an optimized machine learning approach for arm pose estimation from smartwatch sensor data. Key methodological contributions include:
- Multimodal Sensor Fusion: The approach leverages data from a smartwatch's gyroscope, accelerometer, atmospheric pressure sensor, and rotation vector sensor.
- Two-Step Calibration: To mitigate sensor variability and account for body orientation, a novel two-step calibration procedure is introduced. This involves calibrating atmospheric pressure and establishing a consistent forward-facing body direction.
- Multimodal Pose Prediction: Instead of predicting a single arm pose, the model outputs a distribution of possible wrist and elbow positions. This allows for uncertainty estimation and the detection of multiple plausible arm postures.
- Network Architectures: Both feedforward and Long Short-Term Memory (LSTM) networks were investigated. The LSTM architecture, which can process sequential data, generally outperformed the feedforward network.
- Prediction Targets: Various representations for prediction targets were explored, including polar and Cartesian coordinates for positions, and 6-Dimensional Rotation Representation (6DRR) and quaternions for rotations. The 6DRR representation, combined with the constraint of known arm lengths, yielded the best results.
- Activation Function: The Scaled Exponential Linear Unit (SELU) activation function was employed to induce self-normalizing properties and improve robustness to outliers.
- Monte Carlo Dropout: To handle multimodal pose distributions, Monte Carlo (MC) dropout was used during prediction to generate multiple stochastic forward passes, resulting in a distribution of outputs.
- Speech Recognition Integration: Microphone data from the smartwatch is transcribed into voice commands, further enhancing the control interface.
Datasets
The authors collected a new dataset for training and evaluation. This dataset consists of:
- Ground Truth: Upper-body motion capture data from an OptiTrack system, recorded at approximately 120 Hz, with 25 markers on a suit worn by participants.
- Smartwatch Data: Sensor measurements (gyroscope, accelerometer, atmospheric pressure, rotation vector) streamed from a Samsung Galaxy Watch 5 at approximately 50 Hz.
- Data Collection: Participants performed various arm movements while wearing the smartwatch on their left wrist. The data collection was conducted in accordance with Arizona State University (ASU) guidelines, with written informed consent obtained.
Key Insights and Ideas
- Smartwatches as Powerful Interfaces: The paper demonstrates that a single, readily available smartwatch can be a surprisingly powerful sensor for nuanced human motion tracking, challenging the notion that only high-end systems suffice.
- The Power of Uncertainty: Predicting a distribution of poses, rather than a single point estimate, is crucial. This "multimodality" allows for more robust decision-making, especially when sensor data is ambiguous, and opens doors for selecting safer or more optimal robot actions.
- Calibration is Key: The simple two-step calibration procedure significantly improves the accuracy of the smartwatch data by accounting for environmental factors (pressure) and user orientation.
- 6DRR for Rotations: The 6-Dimensional Rotation Representation (6DRR) proved to be a more effective target for training neural networks for rotation prediction compared to quaternions or Euler angles, especially when combined with known kinematic constraints.
- LSTM's Temporal Advantage: The LSTM architecture's ability to maintain an internal state of past sensor readings provides a significant advantage for time-series data like arm movements.
Key Results
- Accuracy Improvement: The proposed approach achieved a 40% reduction in prediction error compared to previous state-of-the-art methods.
- Mean Error: The model achieved a mean error of 2.56 cm for wrist and elbow positions.
- Specific Errors: The best performing model (LSTM with 6DRR) reported a median error of 2.33 cm for wrist and 1.61 cm for elbow positions.
- Real-time Performance: The system can stream data and perform MC dropout predictions at approximately 40 Hz on a reasonably equipped system.
- Usability Demonstrations: Two use-cases were demonstrated:
- Intervention Task: Users could effectively intervene in a robot's task (e.g., stopping it mid-action, redirecting an object) using voice commands and arm gestures.
- Learning Task: The system enabled swift data collection for training robot policies via imitation learning, with trajectories collected in under two minutes.
Limitations and Future Work
- Calibration Dependency: The current approach requires a two-step calibration procedure whenever the user's body orientation or location changes significantly.
- Fast/Unergonomic Movements: Rapid wrist rotations or unergonomic arm motions can still affect prediction accuracy, sometimes leading to bimodal distributions where the model struggles to pinpoint a single pose.
- Future Directions:
- Adding a Second IMU: Incorporating a second IMU could alleviate the calibration dependency and allow for more freedom of movement.
- Smartphone Integration: Leveraging the connected smartphone as a second IMU is a promising avenue.
- Handling Multimodality: Further developing methods to dynamically select the most likely pose from multimodal distributions using additional cost functions or predictive models.
- Improving Robustness: Enhancing the system's ability to handle fast rotations and unergonomic movements.
Summary updated