Learning from Sparse and Incomplete Electronic Health Records

199 concepts176 papers4 proposals

Medical records are not designed for machine learning. As Simon Schallmoser notes in Machine Learning for Predicting Micro- and Macrovascular Complications in Individuals With Prediabetes or Diabetes: Retrospective Cohort Study, Electronic Health Records (EHRs) are optimized for billing and administrative workflows rather than systematic clinical research. Consequently, data sparsity is pervasive. Vital signs are recorded irregularly, laboratory tests are ordered only when clinically indicated (David P. Chen et al.), and critical metrics like body mass index (BMI) or family history are frequently omitted (Kui Jiang et al.). In real-world registries, missingness can exceed 90% for key variables (Robert Kueffner et al.).

Traditional statistical approaches often rely on list-wise deletion—simply discarding any patient record containing a missing value. However, Xuetong Wu et al. demonstrate in Imputation techniques on missing values in breast cancer treatment and fertility data that this practice causes severe information loss, drastically reducing statistical power and introducing bias. To build reliable predictive models, modern machine learning approaches must move beyond deletion, finding ways to intelligently impute, bypass, or compensate for these data gaps.

Borrowing Strength from Patient Cohorts

When an individual patient's longitudinal record is too brief or incomplete, a powerful intuition is to leverage group-level intelligence. If a specific lab value is missing for Patient A, we can infer its likely range or impact by looking at a "neighborhood" of clinically similar patients.

This concept of cohort-based knowledge sharing is formalized by Chaohe Zhang et al. in GRASP: Generic Framework for Health Status Representation Learning Based on Incorporating Knowledge from Similar Patients. The authors utilize Graph Convolutional Networks (GCNs) to link patients into a relational graph, allowing information to flow across similar clinical profiles and fill in individual gaps.

Similarly, Changshuo Liu introduces an interpretable cohort intelligence framework in NeuralCohort: Cohort-aware Neural Representation Learning for Healthcare Analytics, demonstrating that group-level dynamics can effectively compensate for the noise and missingness inherent in single-patient histories. In specialized domains like acute trauma, Tie Bo Wu leverages a hierarchical subspace approach in Understanding Coagulopathy using Multi-view Data in the Presence of Sub-Cohorts: A Hierarchical Subspace Approach to borrow statistical strength across distinct patient sub-cohorts, stabilizing latent variable models even when primary diagnostic views are highly sparse.

Cross-Modal Alignment and View Decoupling

Clinical data sparsity is rarely uniform across all modalities. While structured database fields might lack explicit diagnosis codes or lab values, the missing information often leaves a proxy footprint in unstructured clinical notes, demographic profiles, or molecular biomarkers.

One strategy is to extract structured insights directly from narrative text. Elizabeth W. Karlson et al. demonstrate in Automatic Prediction of Rheumatoid Arthritis Disease Activity from the Electronic Medical Records that natural language processing can unlock critical clinical metrics hidden within unstructured text notes, overcoming the sparsity of structured database fields.

When multiple distinct data streams are available, models can align them to impute missing phenotypes. Qiannan Zhang et al. propose a view-decoupling and knowledge-distillation framework in Unified Insights: Harnessing Multi-modal Data for Phenotype Imputation via View Decoupling, which maps biological data (like proteomics) and clinical data into separate views to filter out noise and impute missing phenotypes.

To fuse these disparate streams without requiring complete feature sets, Chloé Pou-Prom et al. utilize Generalized Canonical Correlation Analysis (GCCA) in Learning multiview embeddings for assessing dementia, constructing robust multi-view embeddings from incomplete linguistic and demographic data. Furthermore, Brian L. Hill et al. show in Methylation risk scores are associated with a collection of phenotypes within electronic health record systems that epigenetic data can serve as a powerful substrate for reconstructing missing laboratory results within EHR systems.

Latent Physiological Modeling and Self-Supervision

Instead of treating missing values as static blanks to be filled with statistical averages, another paradigm treats missingness as a form of data corruption, or views the observed sparse data as a partial reflection of underlying, unobserved physiological states.

Self-supervised architectures are uniquely suited for this task. In Transformer-Based Unsupervised Learning for Early Detection of Sepsis (Student Abstract), Wei Li et al. introduce a Denoising Transformer AutoEncoder (DTAE). By intentionally treating the sparse and noisy medical record as a corrupted signal, the autoencoder learns to capture complex feature correlations and reconstruct a complete clinical picture without requiring explicit labels.

Alternatively, mechanistic biological models can be paired with data assimilation techniques to infer hidden states. Lauren R. Richter et al. demonstrate this in Data assimilation on mechanistic models of glucose metabolism predicts glycemic states in adolescents following bariatric surgery. By applying data assimilation to mathematical models of metabolism, the authors estimate latent physiological traits—such as insulin sensitivity—even when critical specialized laboratory measurements are entirely missing from the patient's EHR.

Compensating with External Knowledge Graphs

When a patient's history is exceptionally brief—such as during the first 24 hours of an intensive care admission—the data may be too sparse for a model to learn complex disease-drug interactions from scratch. To resolve this, researchers inject external, structured medical knowledge to provide a baseline reasoning framework.

In Knowledge-aware Attention Network for Medication Effectiveness Prediction, Xian Wu et al. address this exact limitation. By integrating external biomedical knowledge graphs with a cross-attention mechanism, their network anchors its predictions in established therapeutic mechanisms. This external knowledge acts as a structural scaffold, allowing the model to make accurate clinical predictions even when the individual patient's electronic record provides minimal data coverage.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 162 papers
  2. Version 1 · 2026-10-04 · 277 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_clinical_data_sparsity_c6dce521,
  title = {clinical data sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/clinical-data-sparsity/rdr_c6dce5213d7840ab8d177d4bbc18ea07},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_clinical_data_sparsity_c6dce521_v2,
  title = {clinical data sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/clinical-data-sparsity/rdr_c6dce5213d7840ab8d177d4bbc18ea07/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake