Introduction
Counting repetitive human actions, such as gym repetitions or physical therapy exercises, is a fundamental task for wearable health monitoring. Current approaches generally fall into two categories: class-specific counters, which are limited to pre-trained actions (e.g., walking), and class-agnostic counters, which typically identify and count only the most dominant periodic signal in a data stream.
However, real-world scenarios often require counting specific, non-dominant actions. For instance, a physical therapist might want to count a specific movement error that occurs infrequently amidst other activities. Identifying these "exemplars" in time-series sensor data is challenging; unlike images, where a user can "draw a box," temporal data lacks an intuitive manual labeling interface. This paper introduces a framework to conveniently provide exemplars via vocal cues and a model capable of counting these actions even when they are not the primary signal.
The ExRAC Framework
The proposed system, ExRAC (Exemplar-based Repetition Action Counting), utilizes a multi-stage pipeline to bridge the gap between user intent and automated counting.
The pipeline consists of five key stages:
- Exemplar Identification via Audio: Users specify the action of interest by vocalizing "one," "two," and "three" during the first three repetitions. A pre-trained audio classifier (BC ResNet) combined with a constrained dynamic programming optimization identifies these cues in the temporal stream.
- Feature Embedding: Raw accelerometer and gyroscope data are processed through temporal convolutions to generate per-window embeddings.
- Hybrid Similarity Estimation: The model compares extracted exemplars to the rest of the sequence using a combination of correlation (for general patterns) and Soft-DTW (to handle temporal stretching and variations in speed).
- Exemplar-Infused Feature Embedding: Similarity maps are fused with original sensor embeddings using a Feature Pyramid Network (FPN).
- Density Estimation: The system generates a temporal density map. The final count is obtained by integrating (summing) this map over time.
To maintain the structural integrity of the sensor data during embedding, the authors employ a distance-preserving loss inspired by Laplacian Eigenmaps, ensuring the encoder preserves local neighborhood relationships from the raw signal.
The Diverse Wearable Counting (DWC) Dataset
To evaluate the system, the authors introduced the Diverse Wearable Counting (DWC) dataset. This dataset is uniquely designed to reflect "in the wild" conditions, featuring synchronized audio and inertial sensor data across a wide variety of activities.
The DWC dataset includes 1,502 entries from 37 subjects performing 50 different action categories, ranging from kitchen chores to industrial factory tasks. Crucially, it includes "non-dominant" samples where the target action may occupy as little as 10-20% of the total recording, providing a rigorous benchmark for few-shot counting.
Key Results and Insights
ExRAC demonstrates a significant performance leap over existing state-of-the-art (SOTA) class-agnostic models such as RepNet and TransRAC. On the DWC test set, ExRAC achieved a Mean Absolute Error (MAE) of 7.47, a 30% improvement over RepNet (MAE 10.82) and TransRAC (MAE 12.97).
Ablation studies revealed that the constrained audio detection and pre-training on synthetic data were the most impactful components. By mining action templates and synthesizing sequences with varied scaling and distractor actions, the model gained substantial robustness.
Conclusion and Future Work
ExRAC provides a non-intrusive, "voice-as-UI" approach to labeling temporal data, solving the segmentation problem through natural human rhythm. While the system currently depends on clear audio and tight synchronization, it opens new avenues for personalized action monitoring. Future work may explore real-time form correction and handling even more complex "actions within actions" in noisy environments.
Summary updated