MMPF: Multi-Modal Perception Framework for Abnormal Medical Condition Detection

Chuyi Zhong, Dingkang Yang, Peng Zhai, Lihua ZhangAAAI 2025 · DOI

Introduction

As the global population ages, the demand for early detection of sudden medical emergencies—especially for elderly individuals living alone—is skyrocketing. Traditional monitoring relies on two main pillars: professional healthcare staff, which is labor-intensive and expensive, or wearable sensors, which can be noisy, uncomfortable, and often go unworn.

While computer vision offers a non-invasive alternative, existing models have significant blind spots. Most focus on a single condition (like fall detection) or extract broad, global features that overlook subtle but critical "outward manifestations" of distress, such as specific pained facial expressions or protective body postures (e.g., clutching one's chest during a heart attack). The Multi-Modal Perception Framework (MMPF) addresses this by effectively fusing diverse visual cues—face, body, and environment—into a single, reliable diagnostic signal.

The MMPF Architecture

The MMPF processes an image through four distinct streams to capture a comprehensive view of the subject's state:

  1. Whole Image: Captures the general context of the scene.
  2. Facial Expression: Uses MT-CNN for face detection and DeepFace to categorize emotions (Anger, Fear, Sadness, etc.) as proxies for physical distress.
  3. 2D Posture: Employs a lightweight version of OpenPose to map skeletal keypoints.
  4. 3D Posture: Uses a "Lifting Network" to project 2D coordinates into 3D space, providing depth information that helps the model understand body orientation even when parts of the torso are occluded.
Overview of the MMPF architecture showing the four-stream feature extraction and the Cascade Squeeze-Excitation fusion module.

The core innovation of the framework is the Cascade Squeeze-Excitation (CSE) module. Standard Squeeze-Excitation (SE) blocks help models focus on important data channels, but they can be susceptible to noise. The CSE improves on this with two stages: Adaptive Squeeze-Excitation (ASE), which refines features within a single modality using a query-key-value attention mechanism, and Multi-modal Squeeze-Excitation (MSE), which fuses features across different modalities by dynamically weighing which stream (face vs. pose) is most informative for a given frame.

Robustness through Random Joint Occlusion

To ensure the model performs in real-world home environments where furniture or lighting might block a camera's view, the researchers introduced a Random Joint Occlusion (RJO) module during training. This module intentionally "hides" certain body joints, forcing the model to become robust against partial occlusions.

Example of a heart infarction label with Random Joint Occlusion (RJO) applied to simulate real-world visibility challenges.

Experimental Results and Insights

The researchers developed and released the MMPF Dataset, containing 10,500 images covering four abnormal conditions (Infarction, Headache, Stomach Pain, and Falls) and three normal activities (Reading, Sweeping, and Eating).

In comparative testing, the MMPF framework demonstrated clear superiority over previous methods:

  • Accuracy: On the MMPF dataset, the framework achieved 88.13% accuracy, significantly outperforming the previous state-of-the-art (82.41%).
  • Cross-Dataset Validation: On the NTU RGB+D dataset, the model maintained its lead with 82.51% accuracy.
  • Efficiency: By utilizing "Lightweight OpenPose" and efficient backbone networks like MobileNetV2, the system remains viable for edge devices or mobile platforms.

Ablation studies revealed that while facial expressions are helpful, body posture often conveys more relevant information for specific conditions, such as the "bent-over" posture associated with stomach pain. Furthermore, moving from 2D to 3D pose estimation significantly boosted robustness by allowing the model to better interpret "foreshortened" poses where a person is leaning toward or away from the camera.

Conclusion

MMPF represents a significant step forward in non-invasive medical monitoring. By combining facial analysis with 3D skeletal data through a sophisticated fusion module, it achieves state-of-the-art accuracy without requiring wearable sensors. Future work will focus on expanding dataset diversity and integrating temporal dynamics through video-based analysis to better distinguish between sudden emergencies and normal rapid movements.

Summary updated

BibTeX
@inproceedings{zhong2025mmpf,
  author = {Chuyi Zhong and Dingkang Yang and Peng Zhai and Lihua Zhang},
  title = {{MMPF: Multi-Modal Perception Framework for Abnormal Medical Condition Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i10.33157},
}
Figure