Rethinking Multimodality: Optimizing Multimodal Deep Learning for Biomedical Signal Classification

Timothy Oladunni, Alex Wong

Abstract

This study proposes a novel perspective on multimodal deep learning for biomedical signal classification, systematically analyzing how complementary feature domains impact model performance. While fusing multiple domains often presumes enhanced accuracy, this work demonstrates that adding modalities can yield diminishing returns, as not all fusions are inherently advantageous. To validate this, five deep learning models were designed, developed, and rigorously evaluated: three unimodal (1D-CNN for time, 2D-CNN for time-frequency, and 1D-CNN-Transformer for frequency) and two multimodal (Hybrid 1, which fuses 1D-CNN and 2D-CNN; Hybrid 2, which combines 1D-CNN, 2D-CNN, and a Transformer). For ECG classification, bootstrapping and Bayesian inference revealed that Hybrid 1 consistently outperformed the 2D-CNN baseline across all metrics (p-values < 0.05, Bayesian probabilities > 0.90), confirming the synergistic complementarity of the time and time-frequency domains. Conversely, Hybrid 2's inclusion of the frequency domain offered no further improvement and sometimes a marginal decline, indicating representational redundancy; a phenomenon further substantiated by a targeted ablation study. This research redefines a fundamental principle of multimodal design in biomedical signal analysis. We demonstrate that optimal domain fusion isn't about the number of modalities, but the quality of their inherent complementarity. This paradigm-shifting concept moves beyond purely heuristic feature selection. Our novel theoretical contribution, "Complementary Feature Domains in Multimodal ECG Deep Learning," presents a mathematically quantifiable framework for identifying ideal domain combinations, demonstrating that optimal multimodal performance arises from the intrinsic information-theoretic complementarity among fused domains.

Introduction

In the field of biomedical signal processing, the prevailing paradigm for multimodal deep learning has long been "the more, the merrier." Researchers frequently assume that integrating data from as many domains as possible—such as time, frequency, and time-frequency—will inherently lead to more robust and accurate models. However, this approach often overlooks the risks of representational redundancy, increased model complexity, and overfitting.

This paper challenges the conventional wisdom by investigating the trade-offs between model complexity and classification performance in Electrocardiogram (ECG) analysis. We introduce a framework for "Complementary Feature Domains," arguing that the success of multimodal fusion depends on the intrinsic information-theoretic synergy between domains rather than the sheer volume of data sources.

Methodology

To test our hypothesis, we developed a rigorous comparative study involving five distinct neural architectures designed to extract features from different signal domains. The workflow follows a structured five-step process, from raw signal acquisition to final performance analysis.

Proposed Multimodal Classification Framework

Architectural Design

  1. Unimodal Models:
    • 1D-CNN: Designed to capture temporal dependencies directly from ECG waveforms.
  2. 2D-CNN: Utilizes scalograms to extract time-frequency features.
  3. 1D-CNN-Transformer: Focuses on frequency-domain features, leveraging the Transformer's ability to extract long-range dependencies.
  4. Multimodal Models:
    • Hybrid 1: An intermediate fusion model combining the 1D-CNN (Time) and 2D-CNN (Time-Frequency) branches.
  5. Hybrid 2: A more complex fusion integrating all three branches: Time, Time-Frequency, and Frequency.

Data Augmentation and Preprocessing

The study utilized a digitally recorded ECG dataset from Mendeley Data, covering four diagnostic classes: Myocardial Infarction (MI), Abnormal Heartbeat (Arrhythmias), History of MI, and Normal rhythm. To address class imbalance, we employed ADASYN (Adaptive Synthetic Sampling) to generate physiologically plausible synthetic samples for rarer conditions.

Intra-Class Variance of Real vs. Synthetic Data

As shown in the radar chart above, the synthetic data generated via ADASYN maintains a variance profile highly consistent with real-world samples across all four classes, ensuring that the model learns robust features without being biased by class frequency.

Results and Discussion

The experimental results provided a striking refutation of the "more is better" assumption.

  • The Peak Performance: Hybrid 1 emerged as the top performer, achieving a peak 96% accuracy and a 95.7% F1-score. Statistical validation via non-parametric bootstrapping and Bayesian inference confirmed that Hybrid 1's gains over unimodal baselines were statistically significant (p < 0.05).
  • The Redundancy Trap: Surprisingly, Hybrid 2—which included the frequency-domain Transformer—saw a performance drop to 94% accuracy. An ablation study confirmed that the frequency domain introduced redundant information that overlapped with the time-frequency features, effectively "diluting" the signal and increasing noise.
  • Quantifying Complementarity: Using Mutual Information (MI) analysis, we found that while linear correlations between domains were low, non-linear dependencies were present. Hybrid 1 succeeded because the Time and Time-Frequency domains provided high individual predictive power with low mutual redundancy, creating a synergistic effect.

Conclusion

This research redefines the design principles for multimodal biomedical signal analysis. By demonstrating that the inclusion of redundant domains can actually degrade performance, we advocate for a more selective, information-theoretic approach to feature fusion. The "Complementary Feature Domains" framework provides a path toward more efficient, accurate, and interpretable AI models in clinical diagnostics, proving that in deep learning, the "Goldilocks" zone of complexity is often superior to maximalist architectures.

Summary updated

Abstract

This study proposes a novel perspective on multimodal deep learning for biomedical signal classification, systematically analyzing how complementary feature domains impact model performance. While fusing multiple domains often presumes enhanced accuracy, this work demonstrates that adding modalities can yield diminishing returns, as not all fusions are inherently advantageous. To validate this, five deep learning models were designed, developed, and rigorously evaluated: three unimodal (1D-CNN for time, 2D-CNN for time-frequency, and 1D-CNN-Transformer for frequency) and two multimodal (Hybrid 1, which fuses 1D-CNN and 2D-CNN; Hybrid 2, which combines 1D-CNN, 2D-CNN, and a Transformer). For ECG classification, bootstrapping and Bayesian inference revealed that Hybrid 1 consistently outperformed the 2D-CNN baseline across all metrics (p-values &lt; 0.05, Bayesian probabilities &gt; 0.90), confirming the synergistic complementarity of the time and time-frequency domains. Conversely, Hybrid 2's inclusion of the frequency domain offered no further improvement and sometimes a marginal decline, indicating representational redundancy; a phenomenon further substantiated by a targeted ablation study. This research redefines a fundamental principle of multimodal design in biomedical signal analysis. We demonstrate that optimal domain fusion isn't about the number of modalities, but the quality of their inherent complementarity. This paradigm-shifting concept moves beyond purely heuristic feature selection. Our novel theoretical contribution, "Complementary Feature Domains in Multimodal ECG Deep Learning," presents a mathematically quantifiable framework for identifying ideal domain combinations, demonstrating that optimal multimodal performance arises from the intrinsic information-theoretic complementarity among fused domains.
BibTeX
@inproceedings{oladunni2025rethinking,
  author = {Timothy Oladunni and Alex Wong},
  title = {{Rethinking Multimodality: Optimizing Multimodal Deep Learning for Biomedical Signal Classification}},
  booktitle = {IEEE Access},
  year = {2025},
  doi = {10.1109/access.2025.3605315},
}
Figure