Abstract
The COVID-19 pandemic exposed a global deficiency in delivering systematic, data-driven guidance to protect citizens and coordinate vaccination programs. At the same time, medical histories are routinely recorded in most healthcare systems and are instantly available for risk assessment. Here, we demonstrate the utility of medical history in determining the risk for 1,883 diseases across clinical specialties and facilitating the rapid response to emerging health threats, exemplified by COVID-19. We developed a neural network to learn disease-specific risk states from routinely collected health records of 502,460 UK Biobank participants, demonstrating risk stratification for nearly all conditions, and validated this model on 229,830 individuals from the All of Us cohort. When integrated into Cox Proportional Hazard Models, we observed significant discriminative improvements over basic demographic predictors for 1,774 (94.3%) endpoints. After transferring the unmodified risk models to the All of Us cohort, the discriminate improvements were replicated for 1,347 (89.8%) of 1,500 investigated endpoints, demonstrating model generalizability across healthcare systems and historically underrepresented groups. Our study demonstrates the potential of medical history to rapidly respond to emerging health threats by systematically estimating risk for thousands of diseases at once at minimal cost.
Introduction
Modern clinical practice is predominantly reactive, focusing on treating diseases after they manifest. While "bespoke" risk scores exist for high-profile conditions like cardiovascular disease or breast cancer, the vast majority of human diseases lack systematic risk-stratification tools. Furthermore, existing scores often require specific laboratory measurements or physical exams, creating barriers to routine clinical use. Conversely, healthcare systems possess vast amounts of longitudinal Electronic Health Record (EHR) data—including diagnoses, medications, and procedures—that exceed human cognitive limits for comprehensive risk assessment. This study investigates whether a unified machine-learning model can leverage this "medical history" to provide a comprehensive, data-driven risk profile for nearly every major disease category.
Methodology
The researchers developed a multi-task neural network, specifically a multi-layer perceptron (MLP), to learn "risk states" from routinely collected health data.
Medical records were mapped to the OMOP Common Data Model, representing each individual as a 15,595-dimensional binary vector indicating the presence or absence of specific drugs, conditions, or procedures prior to recruitment. The model architecture consists of three fully connected layers with 4,096 hidden units each, utilizing layer normalization, dropout, and leaky ReLU activations. Instead of training 1,883 separate models, the network predicts log partial hazards for all endpoints simultaneously, allowing it to extract shared features across related diseases. These neural-network-derived "risk states" were then integrated into Cox Proportional Hazard (CPH) models to generate absolute 10-year risk estimates, adjusting for age and biological sex.
The primary cohort for training and internal spatial validation was the UK Biobank (UKB), comprising 502,460 individuals. For external validation, the researchers used the All of Us Research Program, a diverse US-based cohort of 229,830 individuals, including higher proportions of groups historically underrepresented in biomedical research.
Results and Validation
The medical history model provided significant discriminative improvements over basic demographics for 1,800 (95.6%) of the investigated endpoints in the UK Biobank. Stratification was particularly strong: for 74.6% of diseases, individuals in the top 10% of predicted risk had over 10 times as many events as those in the bottom 10%. For certain conditions, such as abdominal aortic aneurysm, this ratio reached 212x. In cardiovascular disease, the history-only model performed competitively with established clinical scores like QRISK3 and ASCVD, despite not requiring lab tests.
A key finding was the model's "zero-shot" generalizability. Models trained on UK data were transferred to the US "All of Us" cohort without modification. Despite differences in healthcare systems and coding practices, the model improved predictions for 83.5% of endpoints in the US cohort. Furthermore, the model demonstrated an ability to infer disease onset from indirect data, such as using psoriasis medication prescriptions to predict future formal diagnoses. The researchers also found that "rare" medical concepts (appearing in <1% of the population) were highly informative; 91.7% of individuals have at least one such record, often associated with high incident event rates.
To demonstrate agility during health crises, the team combined existing risk states for pneumonia, sepsis, and all-cause mortality to create a "COVID-19 severity score" using pre-pandemic data. This repurposed score significantly outperformed age-based models in identifying individuals at high risk of mortality from the virus.
Discussion
While promising, the use of EHR data is subject to systemic biases, reflecting interactions with healthcare systems rather than pure biological reality. Socio-economic and geographic factors can influence the density of medical records. Additionally, the "healthy volunteer effect" in research cohorts may lead to an underestimation of absolute risk in the general population. Future implementation must address data privacy, governance, and clinical integration to ensure these phenome-wide profiles support medical decision-making without overwhelming providers. This study underscores the underused potential of medical history to systematically estimate risk for thousands of diseases at once at minimal cost, facilitating rapid responses to emerging health threats.
Summary updated