TL;DR
This paper tackles the challenge of insufficient training data for machine learning models that estimate human biomechanics from wearable sensors. They introduce a method to generate realistic simulated inertial sensor data using biomechanical models, which, when combined with real data, significantly improves the accuracy of estimating joint angles and offers moderate improvements for joint moments and ground reaction forces.
What Problem Does This Paper Address?
Accurately assessing human movement biomechanics (like joint angles, forces, and moments) is crucial for applications ranging from sports performance analysis to medical diagnostics and rehabilitation. Wearable inertial sensors (IMUs) offer a convenient way to collect this data outside of lab settings. However, training robust machine learning models, particularly deep learning models like Convolutional Neural Networks (CNNs), requires large, representative datasets. Acquiring such datasets, which involve synchronized IMU and high-fidelity motion capture data, is time-consuming, expensive, and often infeasible. This data scarcity leads to poor generalization of models to unseen movements or individuals.
Methods
The core innovation of this paper lies in their approach to data augmentation through biomechanical simulation. They leverage a physics-based musculoskeletal model to generate synthetic inertial sensor data that corresponds to realistic biomechanical outputs. Here's a breakdown:
- Biomechanical Simulation: They used a planar musculoskeletal model with seven rigid segments and 16 Hill-type muscles. This model was scaled to individual subjects and then used to solve optimal control problems. The goal was to make the simulated model follow randomly sampled joint angle and ground reaction force (GRF) trajectories from a measured dataset, while minimizing muscular effort. This process ensures the simulated movements are physically plausible and dynamically consistent.
- Virtual Inertial Sensor Data: From these simulated biomechanical movements, they generated corresponding virtual inertial sensor data (accelerometer and gyroscope readings) by placing virtual sensors on the simulated model at the same locations as the real sensors.
- Convolutional Neural Networks (CNNs): They employed CNNs to learn the mapping between the inertial sensor data (input) and the biomechanical variables (output). Specifically, they trained eight separate CNNs, each dedicated to estimating a single biomechanical variable (e.g., hip joint angle, knee joint moment, vertical GRF). The network architecture involved two convolutional layers for feature extraction, followed by dense layers for regression.
- Data Combination: The key experimental setup involved training CNNs on:
- Measured data only.
- Measured data augmented with varying amounts of simulated data.
Datasets
- Measured Data: The study utilized a dataset previously collected by Dorschky et al. (2019). This dataset comprised synchronized data from seven custom-built inertial sensors (accelerometers and gyroscopes) and an optical motion capture system (16 cameras) with a force plate. The data included walking and running at various speeds from 10 subjects. For training, data from 7 subjects (S04-S10) was used, and data from 3 subjects (S01-S03) was reserved for testing.
- Simulated Data: This is the novel dataset introduced by the authors. It was generated using the biomechanical simulation method described above, based on the measured data of the training subjects. They generated 1,000 simulations per training subject for both walking and running.
Key Insights and Ideas
- Bridging Physics and Data: The paper elegantly combines physics-based biomechanical modeling with data-driven machine learning. This hybrid approach leverages the strengths of both: physics provides realistic constraints and generates data, while machine learning learns complex mappings and enables real-time estimation.
- The Power of Simulated Data: The core insight is that even with a physics-based model that has inherent limitations (e.g., simplified foot model), simulated data can significantly improve the generalization of machine learning models, especially for variables like joint angles. This is crucial for overcoming the data acquisition bottleneck.
- Impact Varies by Variable: The benefit of simulated data is not uniform across all biomechanical variables. It shows a strong positive impact on joint angles but a more modest effect on joint moments and GRFs, suggesting that model inaccuracies play a larger role for these outputs.
- Robustness to Data Scarcity: The study demonstrates that simulated data can be particularly beneficial when the amount of measured training data is limited, helping to maintain or even improve performance.
Key Results
- Joint Angle Improvement: Adding simulated data to the training set led to substantial reductions in the Root Mean Square Error (RMSE) for joint angles. For hip, knee, and ankle joint angles, RMSE decreased by up to 17%, 27%, and 23%, respectively, when training with a significant amount of simulated data.
- Joint Moment and GRF Improvement: Simulated data also improved the estimation of joint moments and GRFs, though to a lesser extent. RMSE for knee and ankle joint moments decreased by up to 6%, and for anterior-posterior and vertical GRFs, by up to 2% and 6%, respectively.
- Pearson Correlation: Pearson correlation coefficients for joint angles were already high (0.96-0.99) and further improved slightly with simulated data. For kinetics, correlations remained high, with some minor improvements.
- Model Limitations: Simulation-aided estimation of joint moments and GRFs was limited by inaccuracies in the biomechanical model. For instance, the hip joint moment estimation was best without simulated data, potentially due to discrepancies between simulated and reference moments.
- Computational Efficiency: While simulations are computationally intensive (mean CPU time of ~3.6 min per simulation), they can generate vast amounts of data much faster than manual data collection.
Limitations and Future Work
- Model Inaccuracies: The authors acknowledge that inaccuracies in their musculoskeletal model (e.g., simplified foot segment, lack of direct tracking of joint moments in the optimal control problem) limited the improvement for joint moments and GRFs.
- Reality Gap: A "reality gap" was observed between simulated and measured inertial sensor data, with simulated data being smoother. The authors suggest future work could explore more sophisticated models (e.g., considering soft tissue effects) or domain adaptation techniques.
- 3D Biomechanics: This study focused on sagittal plane biomechanics. Future work should extend this approach to 3D analysis, which is more complex but offers a more comprehensive understanding of human movement.
- Data Distribution: The performance was sensitive to the distribution of simulated data. Future work could explore more "light-tailed" data distributions than multivariate normals and investigate methods for iteratively generating simulated data during training.
- End-to-End Learning: The authors mention preliminary work on end-to-end learning for domain adaptation, which yielded worse results, suggesting further investigation is needed.
- Sensor Placement: Future work could also explore generating simulated data with varying sensor positions to improve robustness.
Summary updated