Multimodal BVP and Respiration for Automated Pain Recognition with Hankel-FCN

Valentina Becchetti, Muhammad Umar Khan, Stefanos Gkikas, Alessandro Giuseppi, Raul Fernandez RojasarXiv 2026 · arXiv

TL;DR

Researchers have developed a deep learning framework called Hankel-FCN that objectively recognizes pain levels by analyzing Blood Volume Pulse (BVP) and Respiration (RESP) signals. By transforming these physiological signals into structured Hankel matrix embeddings, the model captures complex underlying system dynamics, achieving a 67.09% accuracy in distinguishing between no pain, low pain, and high pain without requiring handcrafted feature engineering.

What Problem Does This Paper Address?

Pain assessment is notoriously difficult because it is inherently subjective. In clinical settings—especially for patients who are sedated, in a coma, or have severe neurological impairments—traditional self-reporting scales (like the 1-10 scale) are useless.

While physiological signals like heart rate and breathing offer an objective window into the autonomic nervous system's response to pain, they are "noisy" and vary significantly between individuals. Most current deep learning models treat these signals as simple 1D time series. The authors argue this misses the bigger picture: physiological responses are part of a coordinated dynamical system. The challenge lies in explicitly modeling these temporal dynamics to move beyond simple sequence processing and achieve more robust, automated pain recognition.

Methods

The core innovation of this paper is the use of the Hankel Operator combined with a Fully Convolutional Network (FCN).

  1. Hankel Matrix Embedding: Instead of feeding raw signals into the network, the authors transform 1D signals into 2D trajectory representations. Based on Takens’ Embedding Theorem, this method uses time-delayed observations to reconstruct the "state-space" of the underlying physiological process. This preserves temporal dependencies across multiple scales.
  2. Dual-Branch FCN: The framework uses two separate FCN branches—one for BVP and one for RESP. This allows the model to learn modality-specific features independently.
  3. Late Fusion: The outputs of these branches are concatenated (fused) and passed to a final classification head to categorize the state as No Pain (NP), Low Pain (LP), or High Pain (HP).
Fig. 2. Representative architecture of the proposed multimodal Hankel-FCN framework. Modality-specific Hankel embeddings and convolutional branches extract temporal features from BVP and RESP signals, which are fused for pain classification (NP, LP, HP).

The team used Optuna for hyperparameter optimization and evaluated the model using nested leave-one-subject-out cross-validation (LOSO-CV) to ensure the results generalize to new, unseen individuals.

Datasets

The study utilized the AI4Pain dataset, which contains recordings from 65 healthy volunteers.

  • Signals: Blood Volume Pulse (BVP) and Respiration (RESP) sampled at 100 Hz.
  • Conditions: Pain was induced via electrical stimuli at two intensity levels (Low and High).
  • Data Volume: A total of 2,340 10-second segments, balanced evenly across the three classes (780 samples each for NP, LP, and HP).
Fig. 1. Filtered and normalized BVP and RESP signals.

Key Insights and Ideas

  • Multimodal is Essential: The study found that neither BVP nor RESP performed particularly well on their own (58.08% and 52.74% accuracy, respectively). However, when combined, accuracy jumped significantly, proving that cardiovascular and respiratory signals provide complementary information about the body's pain response.
  • Dynamics Matter: The Hankel embedding showed a descriptive improvement over standard FCNs (67.09% vs 62.56%). While the statistical evidence was not yet "conclusive" (p = 0.053), the trend suggests that treating physiology as a dynamical system rather than a flat sequence is a promising path forward.
  • The "Low Pain" Hurdle: The model struggled most with the "Low Pain" class. The confusion matrix shows that LP is often misclassified as either NP or HP, likely because physiological responses to mild pain overlap significantly with both baseline states and more intense reactions.

Key Results

The multimodal Hankel-FCN achieved:

  • Accuracy: 67.09%
  • F1-Score: 66.41%
  • Class-Specific Performance: The model was most reliable at identifying No Pain (68.68% True Positive Rate) and High Pain (68.08%), with Low Pain trailing slightly at 64.74%.
Fig. 4. Mean confusion matrix across the 65 outer LOSO folds for three-class pain classification (NP, LP, HP).

Compared to previous studies on the same AI4Pain dataset using different methods (like SVMs or Vision Transformers), the Hankel-FCN performed competitively, even outperforming several established benchmarks that relied on more complex sensing modalities.

Limitations and Future Work

The authors identified several areas for improvement:

  • Dataset Diversity: The framework was tested on a single dataset of healthy volunteers; it needs validation on clinical populations (e.g., hospital patients).
  • Residual Effects: The "No Pain" segments were taken shortly after pain stimulation, meaning residual physiological effects might have influenced the baseline.
  • Interpretability: Like many deep learning models, the specific features the FCN learns from the Hankel matrix are difficult for humans to interpret.
  • Real-time Deployment: Future work will look at reducing computational complexity to allow these models to run in real-time on clinical monitors or "emotion-aware" companion robots.

Summary updated

Abstract

Automated pain recognition is a fundamental challenge for the advancement of clinical care and empathetic human-computer interaction, due to the subjective and dynamic nature of pain. In this study, we propose a deep learning framework for objective pain assessment based on multimodal physiological signals from the AI4Pain dataset. The approach focuses on Blood Volume Pulse (BVP) and Respiration (RESP), capturing complementary aspects of autonomic responses to nociceptive stimulation. To explicitly model their temporal dynamics, the signals are transformed into structured trajectory representations using Hankel matrix embeddings, enabling the learning of dynamical patterns beyond conventional time-series representations. These embeddings are processed by modality-specific Fully Convolutional Networks, and their outputs are integrated through a late fusion strategy for three-class classification (No Pain, Low Pain, High Pain). Experimental results demonstrate that the proposed method achieves an accuracy of 67.09% and an F1-score of 66.41%, providing competitive performance compared to existing approaches, while relying on no handcrafted feature engineering. Multimodal fusion significantly outperformed either modality alone, and the Hankel embedding showed a trend towards further improvement. Further validation is required to establish generalisability to broader populations and clinical settings.
BibTeX
@misc{becchetti2026multimodal,
  author = {Valentina Becchetti and Muhammad Umar Khan and Stefanos Gkikas and Alessandro Giuseppi and Raul Fernandez Rojas},
  title = {{Multimodal BVP and Respiration for Automated Pain Recognition with Hankel-FCN}},
  year = {2026},
  eprint = {2611.01978},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01978},
}
Figure