Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Haiyang Zhou, Yixin Zhao, Yanzhong Liu, Sichao Lu, Xiang An, Qiang Liu2023 · DOI

TL;DR

This paper introduces a novel multi-sensor system for human activity recognition (HAR) that fuses data from a camera and a millimeter-wave radar. By combining these sensors and employing a CNN-LSTM model with data-level fusion, the system achieves significantly improved accuracy in low-light environments, outperforming single-sensor approaches and other fusion methods.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) is crucial for applications like elderly care and home security. However, existing camera-based systems struggle in low-light conditions, leading to decreased accuracy and increased misclassifications. Wearable sensors, while useful, have limitations in battery life and continuous monitoring. This paper addresses the challenge of robust HAR in low-light environments by proposing a multi-sensor fusion approach that leverages the strengths of both cameras and millimeter-wave radar.

Methods

The core of the proposed system is a CNN-LSTM model designed to extract both spatial and temporal features from the fused sensor data. The CNN component is responsible for capturing spatial patterns, while the LSTM handles the temporal dynamics.

The paper explores three data fusion strategies:

  • Data-level fusion: Raw data from both sensors is combined directly. This method aims to preserve maximum information but requires careful synchronization.
  • Feature-level fusion: Features are extracted independently from each sensor's data using separate CNNs, and then these feature maps are combined (either by addition or concatenation).
  • Decision-level fusion: Each sensor's data is processed independently to produce a classification result, and then these individual decisions are fused (e.g., through averaging, weighted averaging, or taking the maximum).

The authors specifically highlight data-level fusion as the most effective for their system.

Datasets

The study utilized a custom dataset collected by the authors, comprising 3000 combinations of radar and camera data sequences. This dataset was recorded under both normal and low-light conditions, capturing five human activities: sitting, squatting, walking, bending, and falling. The dataset was split into an 8:2 ratio for training and testing.

Key Insights and Ideas

  • Synergy of Sensors: Combining a camera with a millimeter-wave radar is highly effective for HAR, especially in challenging low-light conditions where cameras alone fail. The radar's ability to penetrate darkness and its stability in varying light conditions complement the camera's rich visual information.
  • Data-Level Fusion Dominance: For this specific multi-sensor HAR task, data-level fusion proved to be the most effective strategy. It achieved the best balance in accuracy and misclassification rates, suggesting that preserving raw sensor data and fusing it early in the pipeline is crucial for capturing subtle activity nuances.
  • CNN-LSTM for Spatio-Temporal Features: The CNN-LSTM architecture is well-suited for processing the fused sensor data, effectively extracting both spatial (CNN) and temporal (LSTM) features essential for accurate activity recognition.
  • Low-Light HAR is Achievable: The research demonstrates that robust HAR in low-light environments is feasible with the right sensor combination and fusion techniques, opening doors for more reliable monitoring systems.

Key Results

  • The proposed multi-sensor system with data-level fusion significantly improved HAR accuracy in low-light environments compared to camera-only data. Improvements ranged from 19.87% to 26.68% across different fusion methods.
  • The data-level fusion algorithm achieved the best overall balanced performance, with an average recognition accuracy of approximately 95%.
  • The misclassification rate was reduced to 2%~6% with the data-level fusion approach.
  • In low-light conditions, the CNN-LSTM model with radar data outperformed camera data, with accuracies reaching up to 100% for walking.
  • The confusion matrices showed that while camera data in low-light struggled (e.g., sitting accuracy around 63%), the fused radar data achieved much higher accuracy (e.g., sitting accuracy of 96%).

Limitations and Future Work

  • Camera Privacy Concerns: While the system uses radar to mitigate privacy issues, the presence of cameras can still affect user comfort.
  • Training Data Limitations: The model was trained with normal light camera data and tested on low-light data. Training with low-light camera data yielded suboptimal results, suggesting a need for better low-light camera data augmentation or training strategies.
  • Generalization to Elderly Activities: The dataset was collected from young volunteers, and their activities may not fully represent the diverse range of activities and potential fall scenarios of the elderly. Future work should focus on collecting and analyzing data from elderly individuals in real-life settings.
  • Adaptation to Diverse Lighting: Further research is needed to improve the model's ability to generalize and adapt to various lighting conditions, which is critical for practical HAR applications.
  • Future Improvements: The authors plan to enhance feature extraction, data processing, and fusion algorithms, and to collect more diverse human activity data to build larger datasets and improve model robustness.

Summary updated

BibTeX
@misc{zhou2023multi,
  author = {Haiyang Zhou and Yixin Zhao and Yanzhong Liu and Sichao Lu and Xiang An and Qiang Liu},
  title = {{Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System}},
  year = {2023},
  doi = {10.3390/s23104750},
}
Figure