A Multi-target Bayesian Transformer Framework for Predicting Cardiovascular Disease Biomarkers during Pandemics

Trusting Inekwe, Emmanuel Agu, Winnie Mkandawire, Andrés ColubriarXiv 2025 · DOI · arXiv

Abstract

Managing chronic conditions like cardiovascular disease (CVD) requires consistent monitoring of biomarkers, a process severely disrupted by the COVID-19 pandemic. This paper introduces MBT-CB, a novel Multi-target Bayesian Transformer framework designed to predict multiple CVD biomarkers—LDL cholesterol, HbA1c, BMI, and blood pressure—simultaneously from electronic health records (EHR). By combining pre-trained clinical language models with Bayesian uncertainty estimation and multi-target regression, the system provides robust, "risk-aware" predictions that account for healthcare disruptions and lifestyle shifts. MBT-CB significantly outperforms traditional machine learning and standard transformer baselines, demonstrating the value of joint learning and uncertainty quantification in high-stakes clinical settings.

Introduction

The COVID-19 pandemic introduced significant challenges to the management of chronic diseases. Delayed medical care and radical lifestyle changes disrupted the consistent monitoring of critical CVD biomarkers. Predicting these values from EHR data is complicated by irregular visit intervals, physiological interdependencies between biomarkers, and inherent data noise. Standard regression models often fail to account for heteroscedasticity, where the variance of residuals changes across the range of fitted values, leading to unreliable predictions.

Residuals vs. Fitted Values illustrating heteroscedasticity in biomarker data.

Methodology

The proposed Multi-target Bayesian Transformer for CVD Biomarkers (MBT-CB) architecture integrates several advanced techniques to address these challenges. The framework utilizes ClinicalBERT, a transformer pre-trained on 1.2 billion words of clinical text, to process numerical EHR data formatted as structured "sentences."

To handle uncertainty, the model employs Bayesian Variational Inference (BVI) within the attention mechanism, replacing fixed weights with probability distributions. This allows the model to quantify both aleatoric (data noise) and epistemic (model knowledge) uncertainty. Furthermore, a Deep Multi-Target Regression (DeepMTR) head enables the model to learn shared physiological representations across multiple biomarkers simultaneously. The framework also incorporates temporal and pandemic-specific embeddings to distinguish between pre-pandemic and pandemic-era data patterns.

The MBT-CB framework pipeline, integrating EHR data collection, pandemic context, and the Bayesian Multi-Target Transformer architecture.

Results

The model was evaluated on a dataset of 3,390 records from 304 unique patients in Central Massachusetts, spanning 2019 to 2021. MBT-CB achieved a Mean Absolute Error (MAE) of 0.00887 and a Mean Squared Error (MSE) of 0.00027, significantly outperforming baselines such as Linear Regression, Feed-Forward Neural Networks, and other clinical transformers like MedBERT. Ablation studies confirmed that both the Bayesian components and the multi-target architecture are critical for achieving high performance. The model's ability to produce uncertainty estimates provides clinicians with a measure of confidence, distinguishing between reliable predictions and those influenced by outlier data.

Discussion and Conclusion

MBT-CB demonstrates that joint learning of physiologically linked biomarkers is superior to independent modeling. By harvesting the contextual power of clinical language models for longitudinal numerical data, the framework offers a robust solution for biomarker prediction during periods of systemic disruption. Future work will focus on improving data diversity, enhancing model interpretability for clinicians, and utilizing Network Architecture Search (NAS) to optimize transformer structures for diverse medical tasks.

Summary updated

BibTeX
@misc{inekwe2025multi,
  author = {Trusting Inekwe and Emmanuel Agu and Winnie Mkandawire and Andrés Colubri},
  title = {{A Multi-target Bayesian Transformer Framework for Predicting Cardiovascular Disease Biomarkers during Pandemics}},
  year = {2025},
  eprint = {2509.01794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.01794},
}
Figure