IMU-Based Movement Trajectory Heatmaps for Human Activity Recognition

Orhan Konak, Pit Wegner, Bert Arnrich

TL;DR

This paper proposes a novel method to transform Inertial Measurement Unit (IMU) data into 2D "heatmap" images, which are then used for Human Activity Recognition (HAR). This approach shows promise for improving HAR accuracy, especially in data-scarce scenarios like healthcare, by leveraging powerful image classification techniques.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) using IMUs is valuable for applications like assistive technology and healthcare monitoring. However, a major hurdle is the scarcity of annotated training data, which is particularly costly and time-consuming to acquire in fields like healthcare. Traditional IMU data, while rich in motion information, can be less intuitive for analysis compared to visual data. This paper aims to bridge this gap by converting IMU data into a visual format (heatmaps) that can be processed by advanced image classification models, potentially overcoming data scarcity and enhancing interpretability.

Methods

The core of this work lies in transforming raw IMU data into visual representations. The process involves:

  1. Data Preprocessing: IMU sensor data (acceleration, orientation) are first transformed into a global reference frame and then normalized using a local statistical model.
  2. Pose Estimation: A "deep inertial poser" (DIP) method is employed to estimate a full human body pose from the IMU signals. This uses a skinned multi-person linear model (SMPL) to represent the body.
  3. Trajectory Generation: Specific vertices of the 3D SMPL model are tracked over time to generate 3D trajectories.
  4. Dimensionality Reduction & Heatmap Creation: These 3D trajectories are then reduced to 2D using Principal Component Analysis (PCA). To capture temporal information and highlight frequently traversed paths, these 2D trajectories are converted into 2D "heatmap" images. This step is crucial as it translates the temporal sequence of movements into a spatial representation that convolutional neural networks (CNNs) can process effectively.
  5. Classification: A Convolutional Long Short-Term Memory (ConvLSTM) network is used to classify these heatmap images.

The novelty lies in the specific pipeline of transforming IMU data into these trajectory heatmaps, enabling the application of image-based deep learning techniques to HAR.

Datasets

The primary dataset used for evaluation is DIP-IMU. This dataset is described as the largest publicly available IMU dataset for pose estimation. It contains approximately 90 minutes of IMU data from 10 subjects (9 male, 1 female) wearing 17 IMUs, covering 64 sequences and 330,000 time instants. The data includes motions from five categories: controlled motion of extremities, locomotion, and natural full-body activities, encompassing a total of 13 different activities.

Key Insights and Ideas

  • Visualizing Motion for HAR: Transforming IMU data into trajectory heatmaps provides a visual representation of human activities, making them more accessible for human interpretation and analysis compared to raw time-series data.
  • Leveraging Image Classification for IMU Data: The paper demonstrates that powerful image classification techniques, particularly CNNs, can be effectively applied to HAR by converting sensor data into an image-like format.
  • Addressing Data Scarcity: The heatmap approach shows a performance advantage for smaller datasets, which is a common scenario in specialized domains like healthcare, where data annotation is a bottleneck.
  • Pose Estimation as a Key Enabler: The accuracy of the underlying pose estimation method significantly contributes to the overall HAR performance, especially for smaller subject groups.

Key Results

  • The proposed ConvLSTM model using trajectory heatmaps achieved competitive accuracy, particularly outperforming raw sensor signal-based LSTM models on smaller subsets of subjects.
  • For the full dataset (all 10 subjects), the ConvLSTM model on raw sensor data achieved the highest classification accuracy of 90.59% (±1.55). However, for smaller subsets, the image-based ConvLSTM performed better.
  • The image-based ConvLSTM model showed a performance advantage over models using only SMPL parameters, suggesting that both pose estimation and the heatmap transformation contribute to improved results.
  • Data augmentation techniques (flipping, rotating) applied to the heatmaps did not significantly improve absolute accuracy but tended to reduce confidence intervals, leading to more stable predictions.
  • Confusion matrices revealed that certain activities with similar movements (e.g., "cross stepping" vs. "side stepping," "squats" vs. "leg raises") were more challenging to differentiate, even with the heatmap approach.

Limitations and Future Work

Limitations:

  • Complex Sensor Setup: The method requires a relatively complex setup of six IMU sensors, which is more demanding than using a single smartwatch or smartphone.
  • Information Loss: Converting 3D trajectories to 2D heatmaps results in the loss of some temporal information, specifically direction and speed.
  • Arbitrary Vertex Selection: The choice of which body part vertices to track for heatmap generation is arbitrary and might need to be tailored to specific activities.
  • Transfer Learning: Different sensor manufacturers or setups might require extensive transfer learning.

Future Work:

  • Exploring Pre-trained Models: Investigating the use of pre-trained models (e.g., on MNIST) for heatmap classification.
  • Noise Reduction: Applying noise reduction techniques to IMU data to improve signal quality.
  • Optimal Sensor Location: Further research into the optimal placement of sensors.
  • 3D Image Classification: Exploring the effectiveness of 3D image classification to retain more information.
  • Hybrid Models: Developing hybrid models that combine IMU and image-based feature extraction to leverage the strengths of both approaches, especially for scenarios with both few subjects and abundant data.

Summary updated

BibTeX
@inproceedings{konak2020imu,
  author = {Orhan Konak and Pit Wegner and Bert Arnrich},
  title = {{IMU-Based Movement Trajectory Heatmaps for Human Activity Recognition}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=bwVXu0kl7Z},
}
Figure