Hierarchical Activity Recognition Using Smart Watches and RGB-Depth Cameras

Lei Huang, Zhen Li, Zhiqiang Wei, Lei Huang 0010, Shugang Zhang, Jie NieSensors 2016 · DOI

TL;DR

This paper introduces a novel hierarchical activity recognition system that fuses data from wearable smartwatches and RGB-Depth cameras. It addresses the challenge of multi-person scenarios and dynamic activity sets by proposing an automatic group selection method for the hierarchical structure, leading to more accurate and robust activity recognition.

What Problem Does This Paper Address?

Monitoring personal lifestyle and physical activity is crucial for health evaluation and disease prevention. While various sensors exist, no single solution is perfect. Stationary cameras are useful but limited to indoor, visible areas. Wearable sensors offer continuous monitoring but often provide simplistic data. Combining these sources is promising, but challenges remain: effectively fusing multi-modal data, handling multiple subjects simultaneously, and creating flexible recognition systems that adapt to changing activity sets without manual redesign. Existing hierarchical methods are often hand-designed and lack adaptability.

Methods

The proposed method employs a three-step approach:

  1. Normalized Cross Correlation (NCC)-based Mapping: To address multi-person scenarios, this method binds motion sensor data from a smartwatch to corresponding skeleton data extracted from RGB-D cameras. It calculates the velocity of the hand from both gyroscope (smartwatch) and skeleton (camera) data. By finding optimal thresholds for these velocities, it establishes a correspondence between the two data streams for the same individual.
  2. Feature Extraction:
    • Motion Sensors (Smartwatch): Data from accelerometers and gyroscopes are segmented into 6-second windows. Features extracted include mean, variance, range, spectral energy, and absolute change (AC).
  3. RGB-D Cameras: Skeleton data provides joint positions. Features include 3D Shape Histograms (based on zenith and azimuth angles of joints relative to the hip center) and Edge Histogram Descriptors (EHD) around the hand area.
  4. Hierarchical Classifier with Automatic Group Selection:
    • Two-Layer Structure: The first layer uses motion sensor data to classify activities into groups. The second layer uses image features (from RGB-D cameras) to refine the classification within these groups. This coarse-to-fine approach leverages the strengths of each sensor type.
  5. Automatic Group Selection: Instead of manually designing the hierarchical structure, this method automatically determines the optimal grouping of activities for the first layer. It starts with two groups and iteratively increases the number of groups, evaluating the recognition accuracy for each combination. The group combination that yields the highest accuracy is selected. This allows the system to adapt to new activity sets without manual intervention.

Datasets

  • Custom Dataset: A dataset was collected from ten subjects (six males, four females, average age 32).
  • Indoor Dataset: Subjects performed eight activities (brushing, calling, computer working, drinking, eating, reading, sitting, standing) in front of an RGB-D camera while wearing a smartwatch. Data included 3-axis gyroscope, 3-axis accelerometer, and RGB-D images. Each activity segment was 6 seconds long, with over 400 segments collected per activity.
  • Outdoor/Sensor-Only Dataset: To simulate scenarios where subjects are out of camera view, a separate dataset was generated using only the smartwatch for four activities (walking, standing, running, sitting).
  • Evaluation Protocol: Leave-one-subject-out cross-validation was used for evaluation.

Key Insights and Ideas

  • Synergistic Fusion: Combining wearable motion sensors with RGB-D cameras offers a richer understanding of human activity than either source alone, especially for complex activities.
  • Adaptive Hierarchies: The automatic group selection method is a significant contribution, allowing the hierarchical recognition system to dynamically adapt to changes in the activity set, a crucial feature for real-world applications.
  • Handling Multi-Person Scenarios: The NCC-based mapping is a clever way to disentangle data from multiple individuals, a common challenge in ubiquitous sensing.
  • Smartwatch as a Ubiquitous Sensor: The use of a smartwatch as the primary wearable sensor highlights its potential for continuous, unobtrusive lifestyle monitoring.
  • Activity-Specific Feature Importance: The results suggest that for some activities (e.g., calling), motion sensors are more discriminative, while for others (e.g., computer working, drinking), image features play a more critical role.

Key Results

  • Overall Performance: The proposed hierarchical method achieved an average F1 score of 0.839 across eight activities.
  • Sensor-Only Performance: When only motion sensor data was available (subjects out of camera view), the average F1 score was 0.946 for four simple activities, demonstrating the robustness of motion sensors for basic activity detection.
  • Hierarchical vs. Single Layer: The hierarchical method consistently outperformed a single-layer classifier (using fused features) across all activities, with significant improvements for activities like reading and computer working.
  • Data Fusion Advantage: Combining both motion sensor and image data yielded better results than using either data source alone. The average F1 score for image-only data was higher than for sensor-only data, suggesting image data provides more information for many activities.
  • NCC Mapping Accuracy: The NCC-based mapping method achieved a high accuracy of 92.86% in correctly matching skeleton data to corresponding gyroscope data.

Limitations and Future Work

  • Limitations:
    • The RGB-D camera's range is limited (typically <5m), restricting its use to indoor environments.
  • Skeleton extraction accuracy can be affected by factors like inaccurate hand data shifting, impacting the NCC mapping.
  • Some activities (e.g., drinking and eating) were grouped together in the first layer due to similar motion sensor features, requiring the image layer to differentiate them. The paper notes that differences between these activities from skeleton data were "too slight."
    • The computational complexity of the initial group selection method was high, though a more efficient approach was proposed.
  • Future Work: The paper suggests that the proposed general hierarchical method, which automatically generates the recognition strategy from multi-source data, can be further explored and applied to more complex scenarios. The adaptive nature of the hierarchical structure is a key direction for future research.

Summary updated

BibTeX
@article{huang2016hierarchical,
  author = {Lei Huang and Zhen Li and Zhiqiang Wei and Lei Huang and Shugang Zhang and Jie Nie},
  title = {{Hierarchical Activity Recognition Using Smart Watches and RGB-Depth Cameras}},
  journal = {Sensors},
  year = {2016},
  doi = {10.3390/s16101713},
}
Figure