TL;DR
This paper introduces an end-to-end deep learning system for recognizing complex physical exercises and counting repetitions using only off-the-shelf smartwatches. Achieving near-perfect accuracy for exercise recognition and impressive results for repetition counting, this work offers a practical solution for automated fitness tracking.
What Problem Does This Paper Address?
The paper tackles the challenge of accurately recognizing complex physical exercises and counting repetitions, a crucial aspect of human activity recognition (HAR) for fitness applications. While existing wearable devices can track simple, long-duration activities like running, they struggle with the fast-paced, multi-joint movements common in sports like CrossFit. Manually tracking workouts is tedious, and existing automated solutions often require specialized hardware or extensive feature engineering. The authors aim to bridge this gap by developing a robust, deep learning-based system that can be implemented using readily available smartwatches.
Methods
The core of the proposed solution is an end-to-end deep learning pipeline. For exercise recognition, they employ a Convolutional Neural Network (CNN) that takes windows of raw sensor data (accelerometer, gyroscope, and orientation) from one or two smartwatches as input. This network outputs probability distributions over the 10 predefined exercises.
For repetition counting, they leverage the same CNN architecture but train separate models for each exercise. These models are trained to detect the start of a repetition, signaled by a vibration from the smartwatch during data collection. The output is a binary sequence indicating repetition starts, which is then smoothed to derive the final count. A key aspect of their approach is that it requires no manual feature engineering, relying solely on raw sensor data and a deep learning model.
Datasets
The authors collected a novel dataset specifically for this research due to the lack of publicly available CrossFit sensor data.
- Dataset: Inertial sensor data from two smartwatches (one wrist, one ankle) worn by 54 participants performing 10 complex CrossFit exercises.
- Collection Method: Two modes were used:
- Constrained Workout: Participants performed exercises with a vibration signal from the wrist smartwatch indicating the start of each repetition, facilitating automatic labeling.
- Unconstrained Workout: Participants performed exercises freely without vibration cues, simulating real-world usage.
- Data Size: Approximately 230 minutes of data containing 5,461 repetitions.
- Null Class Data: Collected from participants sitting, standing, and walking to help the model distinguish between exercises and non-exercise periods.
Key Insights and Ideas
- Smartwatches are Sufficient: The study demonstrates that off-the-shelf smartwatches, even a single one worn on the wrist, can achieve very high accuracy for recognizing most exercises.
- Ankle Watch Boosts Accuracy: While a wrist-worn watch is often sufficient, adding an ankle-worn watch significantly improves the recognition of exercises with similar arm movements (e.g., kettlebell thruster vs. press).
- No Feature Engineering Needed: The deep learning approach eliminates the need for manual feature extraction, simplifying the pipeline and potentially making it more adaptable.
- Unified Architecture: Using the same CNN architecture for both recognition and counting streamlines the system and highlights the versatility of the chosen model.
- Constrained Data Collection is Effective: The vibration-based labeling method significantly simplifies data acquisition and enables large-scale, crowd-sourced data collection for future HAR research.
Key Results
- Exercise Recognition Accuracy: Achieved a near-perfect 99.96% accuracy on the constrained workout data using both smartwatches. With only a wrist-worn smartwatch, accuracy was still high at 98.91%.
- Repetition Counting Accuracy: Correctly counted repetitions within an error of ±1 in 91% of the performed sets.
- Sensor Importance: The accelerometer provided the most information, with the gyroscope significantly improving accuracy. The orientation sensor had minimal impact.
- Unconstrained Workout Performance: The model generalized well to unconstrained workouts, correctly recognizing most exercises even without explicit repetition cues.
Limitations and Future Work
- Constrained vs. Unconstrained Generalization: Repetition counting performance decreased for exercises performed in rapid succession in unconstrained workouts compared to the constrained setting. The authors plan to address this by annotating unconstrained data with video recordings and investigating the impact of constrained data collection on generalization.
- Data Collection: While the constrained data collection method is efficient, it might introduce a bias due to participants waiting for vibration cues. Future work aims to mitigate negative effects and improve data collection strategies.
- Online Implementation: The current system operates offline. Future work includes adapting the methods for efficient online computation on low-power devices like smartwatches, which involves addressing engineering challenges related to limited memory and computation resources.
- Crowdsourcing: The authors plan to publish an app to crowdsource data collection, further expanding the dataset and improving model robustness.
Summary updated