Predicting Human Motion Signals Using Modern Deep Learning Techniques and Smartphone Sensors

Taehwan Kim, Jeongho Park, Juwon Lee, Jooyoung Park2021 · DOI

TL;DR

This paper introduces a novel method for predicting human motion signals using a Fourier Neural Operator (FNO) combined with recurrence plots derived from smartphone sensor data. The approach demonstrates superior performance compared to traditional Convolutional Neural Networks (CNNs) in predicting future motion, showing promise for applications like early fall detection.

What Problem Does This Paper Address?

The increasing prevalence of smartphones and wearable sensors has opened doors for advanced healthcare applications. A key area of interest is human motion analysis, which can provide valuable insights into health and well-being. Existing methods often rely on converting time-series sensor data into image formats, such as spectrograms or raw plots, for analysis. However, these methods can sometimes lead to information loss or require significant pre-processing. This paper addresses the challenge of accurately and efficiently predicting future human motion signals from smartphone sensor data, aiming to improve upon existing techniques and enable timely interventions, such as fall detection.

Methods

The core of this research lies in the combination of two key techniques:

  • Recurrence Plots (RPs): RPs are used to transform time-series sensor data into a 2D image format. Unlike traditional RPs that might use thresholds and step functions (which can cause information loss), this method utilizes the distance component of the recurrence plot without these steps. This approach captures both local and global temporal patterns within the sensor data. For multi-axis sensor data (like from a gyroscope), a cross-recurrence plot (CRP) approach is employed, where data from different axes are combined.
  • Fourier Neural Operator (FNO): The FNO is a novel deep learning architecture that operates in the frequency domain. It's designed to be "mesh-independent," meaning it can generalize to different input resolutions without retraining. This is a significant advantage over traditional CNNs, which are typically sensitive to input size. The FNO is used to predict future recurrence plots based on past ones.
  • Decoder Network: To translate the predicted recurrence plots back into raw sensor signals, a decoder network composed of a Long Short-Term Memory (LSTM) and fully connected layers is used.

The overall pipeline involves acquiring raw sensor data, converting it into recurrence plots, using the FNO to predict future recurrence plots, and then using the decoder to reconstruct the predicted sensor signals.

Datasets

The study utilized data collected from six volunteers performing walking and running motions. The sensor signals were collected using an iPhone XS at 30 Hz for approximately 60 seconds per motion. This data was then converted into recurrence plots for training and testing. For additional experiments on stair climbing, the public MotionSense dataset was used.

Key Insights and Ideas

  • Recurrence Plots as Rich Representations: The paper highlights that recurrence plots, when processed without thresholds and step functions, effectively capture complex temporal dynamics from sensor data, serving as a powerful input for deep learning models.
  • FNO's Mesh-Independence Advantage: The FNO's ability to handle varying input resolutions without retraining is a significant advantage, making it more robust and adaptable than traditional CNNs for this task.
  • Personalized Models for Motion Prediction: The experiments demonstrate that personalized FNO models trained on individual subjects yield significantly lower prediction errors, suggesting that individual-specific models are highly effective for human motion analysis.
  • Early Fall Risk Detection Potential: The ability to predict future motion signals allows for the potential to detect fall risks before an actual fall occurs, offering a crucial window for intervention.

Key Results

  • Superior Prediction Accuracy: The FNO model consistently outperformed a comparable CNN model in predicting human motion signals, as evidenced by lower Mean Squared Error (MSE) values across all subjects and motions. For instance, for subject #1 in walking motion, the FNO achieved an MSE of 0.1577, while the CNN achieved 0.2445.
  • Effective Reconstruction: The decoder network successfully reconstructed raw sensor signals from the predicted recurrence plots, demonstrating the integrity of the entire prediction pipeline.
  • Personalized Model Performance: The study showed that personalized FNO models achieved low MSE values when tested on data from the same subject, but higher values when tested on data from other subjects, reinforcing the effectiveness of personalized models.
  • Fall Detection Feasibility: The research suggests that by predicting future motion, a fall risk can be identified slightly earlier than with methods relying solely on current observational data.

Limitations and Future Work

  • Generalization: The authors acknowledge that the presented models are personalized and may not generalize well to predicting the movements of unseen individuals without further training.
  • Scope of Motions: While walking and running are common activities, the study did not cover a comprehensive range of human motions.
  • Future Work: The authors suggest expanding the scope to include more diverse daily activities and motions, such as stair climbing (which was explored with the MotionSense dataset), to create more robust and generalized models. They also envision future applications in personalized healthcare, exercise programs, and physical therapy, and importantly, in enhancing fall detection systems for vulnerable populations like the elderly.

Summary updated

BibTeX
@misc{kim2021predicting,
  author = {Taehwan Kim and Jeongho Park and Juwon Lee and Jooyoung Park},
  title = {{Predicting Human Motion Signals Using Modern Deep Learning Techniques and Smartphone Sensors}},
  year = {2021},
  doi = {10.3390/s21248270},
}

Loading related content...

Figure