Confidence-Calibrated Human Activity Recognition

Debaditya Roy, Sarunas Girdzijauskas, Serghei Socolovschi2021 · DOI

TL;DR

This paper introduces "Deep Time Ensembles" (DTE), a novel ensembling method for human activity recognition (HAR) using wearable sensors. DTE significantly improves the calibration of deep learning models, ensuring their confidence estimates more accurately reflect the true likelihood of predictions, while also boosting classification performance.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) using wearable sensors is crucial for applications ranging from health monitoring to sports analytics. While deep learning models have achieved high accuracy in classifying activities, they often suffer from poor confidence calibration. This means their predicted probabilities, even when high, don't reliably reflect the true likelihood of the predicted activity. This "overconfidence" can lead to unreliable decision-making, especially in safety-critical systems. Existing HAR methods primarily focus on maximizing accuracy, neglecting the crucial aspect of model reliability through calibrated confidence.

Methods

The core innovation is Deep Time Ensembles (DTE). DTE addresses the limitations of fixed-size time windows used in traditional HAR by employing a novel ensembling strategy:

  • Temporal Sequence Extraction with Varying Window Sizes: Instead of using a single fixed window size to extract temporal features from sensor data, DTE extracts multiple temporal sequences using a set of different window sizes. The rationale is that different activities have different temporal patterns and periodicities, making a single window size sub-optimal for all.
  • Ensembling: For each extracted temporal sequence (derived from a specific window size), a separate neural network model is trained. These individual "expert" models are then ensembled by averaging their softmax outputs. This averaging process softens the overconfident predictions of individual models, leading to a more calibrated probability distribution.

DTE can be applied to any base neural network architecture, and the authors evaluated it with CNN, LSTM, and Convolutional LSTM models.

Datasets

The proposed method was evaluated on four publicly available HAR datasets:

  • PAMAP2: A comprehensive dataset with 12 activities and multivariate sensor data (accelerometer, gyroscope, magnetometer, heart rate).
  • UCI: A dataset featuring 6 activities recorded from smartphones, using triaxial accelerometer and gyroscope.
  • WISDM: A dataset with 6 activities recorded using triaxial accelerometer sensors.
  • Skoda: A dataset capturing 10 manipulative gestures/activities of a factory worker, using 3D accelerometer sensors.

Key Insights and Ideas

  • Activity-Specific Window Sizes Matter: The paper strongly suggests that a single, fixed window size is a poor assumption for heterogeneous HAR datasets. Different activities benefit from different temporal resolutions for feature extraction.
  • Ensembling for Calibration, Not Just Accuracy: While ensembles are known to improve accuracy, DTE specifically leverages them to mitigate overconfidence and achieve better calibration. The averaging of multiple models' outputs is key to this.
  • Reliability is as Important as Accuracy: The work highlights that for practical applications, especially in safety-critical domains, a model's confidence in its prediction is as vital as the prediction itself.
  • DTE is Versatile: The method is architecture-agnostic, meaning it can be applied to various deep learning models, making it a broadly applicable technique.

Key Results

  • Improved Calibration: DTE consistently reduced the Expected Calibration Error (ECE) by at least 40% across all datasets and architectures compared to baseline models.
  • Enhanced Classification: DTE outperformed state-of-the-art classification results on the WISDM, UCI HAR, and PAMAP2 datasets, and performed comparably on the Skoda dataset.
  • Comparison with Temperature Scaling: DTE alone provided better or comparable calibration to temperature scaling (a popular post-processing calibration method), and when combined, it further improved results in some cases.
  • Binwise Calibration: DTE demonstrated improved calibration not just in the high-confidence bins but across all bins, indicating a more robust calibration across the entire prediction spectrum.

Limitations and Future Work

  • Window Size Selection: While DTE uses multiple window sizes, the selection of these sizes is currently empirical and requires experimentation. Future work could explore automatic adaptive selection of window sizes or strong learners within the ensemble.
  • Computational Complexity: Ensembling inherently increases computational cost. Future research could investigate model distillation techniques to create lighter models that retain the ensemble's benefits.
  • Real-time Inference: For resource-constrained environments requiring real-time predictions, the ensemble's latency might be a concern. Lightweight modeling and distillation are suggested solutions.
  • Video-Based HAR: The authors suggest extending DTE to video-frame-based human activity recognition, leveraging spatio-temporal and 3D convolutional architectures.

Summary updated

BibTeX
@misc{roy2021confidence,
  author = {Debaditya Roy and Sarunas Girdzijauskas and Serghei Socolovschi},
  title = {{Confidence-Calibrated Human Activity Recognition}},
  year = {2021},
  doi = {10.3390/s21196566},
}
Figure