WearableMil: An End-to-End Framework for Military Activity Recognition and Performance Monitoring

Barak Gahtan, Shany Funk, Einat Kodesh, Itay Ketko, Tsvi Kuflik, Alex Bronstein

TL;DR

This paper introduces WearableMil, a comprehensive framework for recognizing military activities and monitoring soldier performance using wearable sensor data. It tackles challenges like noisy data and missing information with a novel imputation strategy and a hierarchical deep learning model, achieving high accuracy in both individual and group contexts.

What Problem Does This Paper Address?

Military personnel are highly susceptible to musculoskeletal injuries, which significantly impact operational readiness and incur substantial costs. While wearable technology offers a promising avenue for monitoring and preventing these injuries, existing Human Activity Recognition (HAR) systems struggle with the complexities of military training environments. These challenges include handling continuous, noisy, and incomplete data streams, recognizing diverse activities without predefined sessions, and understanding group dynamics. The paper aims to bridge this gap by providing an end-to-end framework that can effectively process, analyze, and recognize activities from large-scale military training datasets, offering actionable insights for trainers.

Methods

The WearableMil framework employs a multi-stage approach:

  • Data Alignment: Raw sensor data (HR, physical activity, sleep) is unified onto a one-minute time grid.
  • Physiologically-Informed Imputation: A novel rule-based framework leverages HR and physical activity thresholds to impute missing sleep states. This strategy significantly reduces "unknown" sleep states by incorporating domain knowledge about physiological responses.
  • Hierarchical Deep Learning Architecture: A two-tiered model is used, consisting of a bidirectional LSTM for temporal encoding and two specialized classifiers for Level 1 (Sleep, Awake, Activity) and Level 2 (specific military activities) classification.
  • Hierarchical Focal Loss: A custom loss function is developed to balance the contributions of both classification levels, addressing class imbalance and focusing on difficult-to-classify samples.
  • "Smart" Visualization: A novel visualization tool using radar charts allows for real-time comparison of individual performance against group medians across various physiological metrics, aiding in performance assessment and anomaly detection.

Datasets

The study utilizes a dataset collected from 135 soldiers wearing Garmin Forerunner 55 smartwatches over six months, accumulating over 15 million data-minutes. This dataset encompasses HR, physical activity (steps, distance), and sleep data. No new benchmark datasets were introduced; instead, the paper focuses on applying their framework to a real-world military training dataset.

Key Insights and Ideas

  • Physiological Imputation is Crucial: The proposed physiologically-informed imputation strategy dramatically reduces missing sleep data, which is vital for understanding soldier recovery and fatigue.
  • Hierarchical Classification Works: A two-level classification approach effectively captures both broad states (sleep/awake) and specific military activities, offering a more nuanced understanding.
  • Window Size Trade-offs: While longer time windows (45-60 minutes) improve general state classification, they can lead to a decline in detecting fine-grained activities due to the strict majority label criterion and reduced sample counts for shorter events.
  • Individual vs. Group Performance: Temporal splits (within-user generalization) consistently outperform user splits (cross-user generalization), highlighting the challenge of adapting models to unseen individuals and the importance of personalized monitoring.
  • Visualization for Actionable Insights: The radar chart visualization provides an intuitive way for supervisors to quickly identify individuals who deviate from group norms, enabling targeted interventions and improved training management.

Key Results

  • High Accuracy: The framework achieves 93.8% accuracy for Level 1 (Sleep, Awake, Activity) and 74.0% F1-score for Level 2 (specific military activities) in temporal splits.
  • Reduced Unknown States: Missing sleep states were reduced from 40.38% to 3.66% of total time.
  • Improved Wake Detection: Wake detection improved from 2.14% to 45.83% of total time.
  • Temporal Splits Outperform: Temporal splits showed significantly higher performance across all metrics compared to user splits.
  • Optimal Window Size: For Level 2 metrics, a 45-minute window size showed a peak in performance, balancing contextual information and sufficient class samples.

Limitations and Future Work

  • Limitations:
    • Reliance on statistical HR calculations rather than more advanced ML methods for imputation.
  • Fixed window sizes may miss short-duration activities.
  • Technical challenges such as sensor inaccuracies and device removal during specific activities (e.g., "Contact-Combat" had 48.5% accuracy).
  • Generalization to new soldiers (user splits) remains challenging.
  • Future Work:
    • Exploring advanced ML approaches like generative adversarial networks for improved data imputation.
  • Implementing adaptive window sizes to optimize fine-grained recognition.
  • Integrating injury prediction models that leverage activity and sleep patterns.

Summary updated

Abstract

Musculoskeletal injuries during military training significantly impact readiness, making prevention through activity monitoring crucial. While Human Activity Recognition (HAR) using wearable devices offers promising solutions, it faces challenges in processing continuous data streams and recognizing diverse activities without predefined sessions. This paper introduces an end-to-end framework for preprocessing, analyzing, and recognizing activities from wearable data in military training contexts. Using data from 135 soldiers wearing \textit{Garmin--55} smartwatches over six months with over 15 million minutes. We develop a hierarchical deep learning approach that achieves 93.8% accuracy in temporal splits and 83.8% in cross-user evaluation. Our framework addresses missing data through physiologically-informed methods, reducing unknown sleep states from 40.38% to 3.66%. We demonstrate that while longer time windows (45-60 minutes) improve basic state classification, they present trade-offs in detecting fine-grained activities. Additionally, we introduce an intuitive visualization system that enables real-time comparison of individual performance against group metrics across multiple physiological indicators. This approach to activity recognition and performance monitoring provides military trainers with actionable insights for optimizing training programs and preventing injuries.
BibTeX
@inproceedings{gahtan2024wearablemil,
  author = {Barak Gahtan and Shany Funk and Einat Kodesh and Itay Ketko and Tsvi Kuflik and Alex Bronstein},
  title = {{WearableMil: An End-to-End Framework for Military Activity Recognition and Performance Monitoring}},
  booktitle = {ICHI},
  year = {2024},
}
Figure