Count What You Want: Exemplar Identification and Few-Shot Counting of Human Actions in the Wild

Yifeng Huang, Duc Duy Nguyen, Lam Nguyen, Cuong Pham, Minh HoaiAAAI 2024 · DOI

Introduction

Counting repetitive human actions, such as gym repetitions or physical therapy exercises, is a fundamental task for wearable health monitoring. Current approaches generally fall into two categories: class-specific counters, which are limited to pre-trained actions (e.g., walking), and class-agnostic counters, which typically identify and count only the most dominant periodic signal in a data stream.

However, real-world scenarios often require counting specific, non-dominant actions. For instance, a physical therapist might want to count a specific movement error that occurs infrequently amidst other activities. Identifying these "exemplars" in time-series sensor data is challenging; unlike images, where a user can "draw a box," temporal data lacks an intuitive manual labeling interface. This paper introduces a framework to conveniently provide exemplars via vocal cues and a model capable of counting these actions even when they are not the primary signal.

The ExRAC Framework

The proposed system, ExRAC (Exemplar-based Repetition Action Counting), utilizes a multi-stage pipeline to bridge the gap between user intent and automated counting.

Overview of the ExRAC pipeline, showing the flow from audio-based exemplar extraction to temporal density map generation.

The pipeline consists of five key stages:

  1. Exemplar Identification via Audio: Users specify the action of interest by vocalizing "one," "two," and "three" during the first three repetitions. A pre-trained audio classifier (BC ResNet) combined with a constrained dynamic programming optimization identifies these cues in the temporal stream.
  2. Feature Embedding: Raw accelerometer and gyroscope data are processed through temporal convolutions to generate per-window embeddings.
  3. Hybrid Similarity Estimation: The model compares extracted exemplars to the rest of the sequence using a combination of correlation (for general patterns) and Soft-DTW (to handle temporal stretching and variations in speed).
  4. Exemplar-Infused Feature Embedding: Similarity maps are fused with original sensor embeddings using a Feature Pyramid Network (FPN).
  5. Density Estimation: The system generates a temporal density map. The final count is obtained by integrating (summing) this map over time.

To maintain the structural integrity of the sensor data during embedding, the authors employ a distance-preserving loss inspired by Laplacian Eigenmaps, ensuring the encoder preserves local neighborhood relationships from the raw signal.

The Diverse Wearable Counting (DWC) Dataset

To evaluate the system, the authors introduced the Diverse Wearable Counting (DWC) dataset. This dataset is uniquely designed to reflect "in the wild" conditions, featuring synchronized audio and inertial sensor data across a wide variety of activities.

Statistics of the DWC dataset, illustrating the distribution of action categories, repetition counts, and sample durations.

The DWC dataset includes 1,502 entries from 37 subjects performing 50 different action categories, ranging from kitchen chores to industrial factory tasks. Crucially, it includes "non-dominant" samples where the target action may occupy as little as 10-20% of the total recording, providing a rigorous benchmark for few-shot counting.

Key Results and Insights

ExRAC demonstrates a significant performance leap over existing state-of-the-art (SOTA) class-agnostic models such as RepNet and TransRAC. On the DWC test set, ExRAC achieved a Mean Absolute Error (MAE) of 7.47, a 30% improvement over RepNet (MAE 10.82) and TransRAC (MAE 12.97).

Ablation studies revealed that the constrained audio detection and pre-training on synthetic data were the most impactful components. By mining action templates and synthesizing sequences with varied scaling and distractor actions, the model gained substantial robustness.

Conclusion and Future Work

ExRAC provides a non-intrusive, "voice-as-UI" approach to labeling temporal data, solving the segmentation problem through natural human rhythm. While the system currently depends on clear audio and tight synchronization, it opens new avenues for personalized action monitoring. Future work may explore real-time form correction and handling even more complex "actions within actions" in noisy environments.

Summary updated

BibTeX
@inproceedings{huang2024count,
  author = {Yifeng Huang and Duc Duy Nguyen and Lam Nguyen and Cuong Pham and Minh Hoai},
  title = {{Count What You Want: Exemplar Identification and Few-Shot Counting of Human Actions in the Wild}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i9.28869},
}
Figure