Modeling inter-individual differences in ambulatory-based multimodal signals via metric learning: a case study of personalized well-being estimation of healthcare workers

Projna Paromita, Karel Mundnich, Amrutha Nadarajan, Brandon M. Booth, Shrikanth S. Narayanan, Theodora ChaspariFrontiers Digit. Health 2023 · DOI

Introduction

Automated well-being tracking—encompassing the detection of stress, anxiety, and mood—is increasingly vital for healthcare workers operating in high-stakes environments. However, traditional "one-size-fits-all" machine learning models often struggle with inter-individual variability. Physiological and behavioral signals are inherently subjective; for instance, a heart rate that signifies stress in one individual might be a resting rate for another. Furthermore, stable traits such as personality, cognitive ability, and lifestyle influence how these signals manifest.

Standard models often fail to generalize across diverse populations, while fully personalized models frequently lack sufficient longitudinal data per individual to train effectively. This research proposes a middle ground: a personalized metric learning approach that understands individual differences without requiring massive datasets from every new user.

Methodology

The core innovation of this study is the application of Metric Learning implemented via a Siamese Neural Network (SNN). Rather than training a model to predict an absolute value (e.g., a specific stress score), the SNN learns to map multimodal features into a transformed "embedding space." In this space, samples with similar well-being scores are pulled together, while dissimilar ones are pushed apart. This process effectively reduces "person-specific noise" and emphasizes the underlying "behavioral signal."

The researchers evaluated three variations of personalization:

  1. ML-A (Ambulatory): Utilizes only wearable sensor data (Fitbit, audio, smart garments).
  2. ML-AT (Ambulatory + Trait): Incorporates static "trait" data (personality scores, IQ, tobacco use) as additional input features.
  3. ML-C (Clustered): Employs K-means clustering on trait data to group similar participants, training separate metric learning models for each cluster.

To further enhance personalization, the team utilized transfer learning, fine-tuning the models with varying percentages (0–90%) of data from target participants to assess how rapidly the model could adapt to a new individual.

Experimental Setup and Datasets

The study utilized the TILES-2018 dataset, tracking 212 hospital workers over 10 weeks. The final analysis included 139 participants with complete data. Features were drawn from three primary sources:

  • Ambulatory Features: Heart rate, sleep stages, and steps from Fitbit Charge 2; acoustic features from smartphone audio; and breathing/ECG data from OMsignal garments.
  • Trait Features: Big Five personality traits, Shipley IQ tests, tobacco usage, and sleep quality.
  • Ground Truth: Daily "Ecological Momentary Assessments" (EMAs) where workers self-reported anxiety, stress, and positive/negative affect.
Figure 4: Distribution of self-reported well-being scores (1-5 scale) showing a heavy skew toward lower values.

One significant challenge in the dataset was the distribution of ground truth labels. As shown in the figures below, the reports were heavily imbalanced toward low-intensity states, which complicates the detection of extreme stress or anxiety.

Figure 5: Frequency distribution of Negative Affect scores, highlighting the predominance of low-intensity reports in the TILES-2018 dataset.

Key Insights and Results

The proposed metric learning approach yielded several critical findings:

  • Relative Distance over Absolute Values: By modeling the relative distance between samples, the approach proved more robust to outliers and sparse data typical of "in-the-wild" studies.
  • Traits as Anchors: Static traits provided essential context. For example, tobacco use was a major factor in how individuals were clustered, significantly impacting the model's interpretation of dynamic sensor data.
  • Disentangling Identity: The SNN embeddings retained less person-identifiable information than raw features while achieving higher predictive accuracy, suggesting the model successfully filtered out individual baseline differences.

In terms of performance, the ML-AT model (Ambulatory + Traits) was the most effective, significantly outperforming standard feedforward neural networks and domain adaptation baselines. While anxiety remained difficult to predict due to data skew, the models showed a sharp increase in accuracy with as little as 10-20% of a new user's data for fine-tuning.

Discussion and Future Work

While the results are promising, limitations remain. The heavy data skew toward "low stress" states limits the model's ability to detect acute crises. Additionally, the daily temporal resolution of the EMAs may not capture the immediate physiological spikes recorded by sensors. Future work will focus on real-time, finer-resolution predictions and the implementation of differential privacy to further protect sensitive user data in clinical settings.

Summary updated

BibTeX
@article{paromita2023modeling,
  author = {Projna Paromita and Karel Mundnich and Amrutha Nadarajan and Brandon M. Booth and Shrikanth S. Narayanan and Theodora Chaspari},
  title = {{Modeling inter-individual differences in ambulatory-based multimodal signals via metric learning: a case study of personalized well-being estimation of healthcare workers}},
  journal = {Frontiers Digit. Health},
  year = {2023},
  doi = {10.3389/fdgth.2023.1195795},
}
Figure