ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities

Julie Mordacq, Léo Milecki, Maria Vakalopoulou, Steve Oudot, Vicky Kalogeiton

Abstract

Detecting physiological changes—such as acute stress or a pilot’s loss of consciousness—requires the integration of diverse data streams, including video, audio, and various biosignals. However, real-world applications often suffer from missing modalities due to sensor failure or environmental constraints. Furthermore, multimodal models frequently develop an over-reliance on dominant modalities, such as video, leading to catastrophic failure when those sources are unavailable. This paper introduces ADAPT (AnchoreD multimodAl Physiological Transformer), a framework designed for robust physiological change detection. By aligning all sensors to a single "anchor" modality and utilizing a masked transformer for fusion, ADAPT maintains high performance even under significant data loss, achieving state-of-the-art results on the StressID and Loss Of Consciousness (LOC) datasets.

Introduction

In high-stakes environments like fighter jet cockpits or intensive care units, multimodal data collection is standard. Systems monitor video feeds, breathing audio, and biosignals like electrocardiograms (ECG) or electromyography (EMG). While theoretically more data leads to better detection, two practical hurdles persist: missing data and modality bias. Sensors frequently detach, cameras are obstructed, or specific equipment (like a pilot's helmet) precludes certain modalities entirely. Moreover, AI models often "cheat" by over-relying on the most information-rich modality (typically video), causing performance to plummet if that specific sensor fails.

Existing fusion techniques often struggle to scale as the number of sensors increases or require complete data during training—an unrealistic expectation for clinical or field deployment. ADAPT addresses these challenges by creating a modality-agnostic representation that remains functional regardless of which sensors are active.

The ADAPT Framework

ADAPT employs a two-step approach to ensure robustness and scalability:

1. Anchoring (Alignment)

Rather than performing computationally expensive all-to-all comparisons between sensors, ADAPT aligns every modality to a single anchor—specifically, video. Using contrastive learning, the framework ensures that features from 1D signals (like ECG) or audio clips map into the same latent space as the corresponding video. This anchoring strategy allows the system to scale linearly; adding a new sensor only requires aligning it to the established anchor.

2. Masked Multimodal Transformer (Fusion)

Once aligned, these features are processed by a Transformer. To handle missing data during inference, ADAPT utilizes Masked Attention. If a modality is unavailable, its corresponding "slot" in the transformer is simply masked out, allowing the model to focus on the remaining available data without losing structural integrity.

3. Modality Dropout

To prevent the model from becoming overly dependent on the video anchor, the authors implement Modality Dropout during training. By randomly removing modalities during the learning phase, the model is forced to identify correlations across whatever sensors are present. This process effectively transfers "video-level" insights into the biosignal encoders.

Experimental Validation

The model was validated on two challenging physiological datasets:

  • StressID: A dataset for identifying stress using ECG, electrodermal activity (EDA), respiration, audio, and video. This dataset naturally reflects real-world conditions, containing 18% missing video and 46% missing audio data.
  • LOC (Loss Of Consciousness): Collected during aeromedical training for fighter pilots, this dataset tracks consciousness alteration induced by high G-forces. It includes ECG, EMG, breathing sounds, pedal pressure, and visual field tests. Crucially, video is missing in 90% of these observations, providing a rigorous test for multimodal robustness.

Key Insights and Results

The Power of the Anchor

A significant finding was that while human experts struggle to detect loss of consciousness using only biosignals (achieving only ~55% accuracy in case studies), ADAPT learns to extract visual-level indicators from simple heart rate data. By anchoring biosignals to video during training, the model "teaches" the biosignal encoders to recognize patterns that were previously only identifiable through visual observation.

Performance Metrics

  • StressID: ADAPT established a new state-of-the-art, outperforming previous fusion methods by 4% in accuracy and 6% in F1-score.
  • LOC Dataset: ADAPT achieved a True Positive Rate (TPR) of 69.5%, representing a +40% improvement over traditional fusion methods. In safety-critical applications like aviation, maintaining a high TPR is vital, as missing a single fainting event can be fatal.
  • Robustness to Missing Anchors: Even when the video anchor was entirely removed during testing, ADAPT's performance remained stable. This confirms that the anchoring process successfully transferred knowledge from the visual domain to the biomedical signals.

Conclusion and Future Work

ADAPT demonstrates that multimodal models can be both robust to sensor failure and computationally efficient through linear scaling. While the framework currently relies on the quality of the anchor modality during the initial training phase, it opens new doors for clinical monitoring in environments where data is frequently incomplete. Future research will explore applying ADAPT to ICU patient tracking and sleep stage monitoring, as well as investigating the "modality gap" inherent in mapping 1D signals to 3D video data.

Summary updated

Abstract

Multimodality has recently gained attention in the medical domain, where imaging or video modalities may be integrated with biomedical signals or health records. Yet, two challenges remain: balancing the contributions of modalities, especially in cases with a limited amount of data available, and tackling missing modalities. To address both issues, in this paper, we introduce the AnchoreD multimodAl Physiological Transformer (ADAPT), a multimodal, scalable framework with two key components: (i) aligning all modalities in the space of the strongest, richest modality (called anchor) to learn a joint embedding space, and (ii) a Masked Multimodal Transformer, leveraging both inter- and intra-modality correlations while handling missing modalities. We focus on detecting physiological changes in two real-life scenarios: stress in individuals induced by specific triggers and fighter pilots\’{loss} of consciousness induced by g-forces. We validate the generalizability of ADAPT through extensive experiments on two datasets for these tasks, where we set the new state of the art while demonstrating its robustness across various modality scenarios and its high potential for real-life applications. Our code is available at https://github.com/jumdc/ADAPT.git.
BibTeX
@inproceedings{mordacq2024adapt,
  author = {Julie Mordacq and Léo Milecki and Maria Vakalopoulou and Steve Oudot and Vicky Kalogeiton},
  title = {{ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities}},
  booktitle = {MIDL},
  year = {2024},
  url = {https://openreview.net/forum?id=N3z9kznRjx},
}
Figure