Bridging the Granularity Gap in Administrative Healthcare Research

This direction is merged into another direction: administrative data granularity gap. Its last version stays readable and citable here.

Version 1 of 170 papers79 conceptsfrozen 2026-10-04

Large-scale observational research has entered a golden age. By leveraging Electronic Health Records (EHRs), insurance claims, and national registries, researchers can now analyze millions of patient journeys simultaneously. However, a recurring tension exists across this field: the trade-off between breadth (the number of patients) and granularity (the depth of clinical detail for each patient).

While administrative datasets offer unprecedented scale, they often lack the "fine-grained" physiological and lifestyle data necessary to establish true causality. This "granularity gap" creates a persistent risk of residual confounding, where unmeasured variables—rather than the treatments being studied—drive the observed outcomes.

The Problem of Unmeasured Confounding

The primary challenge in using real-world data is that clinicians do not assign treatments randomly; they base decisions on subtle clinical cues. If those cues are not captured in the data, the resulting analysis can be misleading.

For example, Yee-Suk Kim, Yuxi Tian, and Jianxiao Yang noted in Comparative safety and effectiveness of alendronate versus raloxifene in women with osteoporosis that administrative datasets often lack bone mineral density scores. Without this metric, it is impossible to know if a doctor chose a specific drug because the patient’s osteoporosis was more severe, potentially making one drug look less effective than it actually is.

Similarly, Nicholas J. Leeper and colleagues found that evaluating the safety of Cilostazol in Practice-Based Evidence: Profiling the Safety of Cilostazol by Text-Mining of Clinical Notes required navigating "residual confounding" caused by missing physiological metrics like vascular pressure. Without these objective markers, researchers cannot perfectly balance treatment and control groups.

The Missing Lifestyle Layer

Beyond physiological metrics, a major blind spot in large-scale data is the "lifestyle layer"—factors like diet, smoking status, and physical activity. These variables are powerful predictors of health but are rarely coded in standardized billing data.

In studies of diabetes and hypertension, this absence is particularly acute. Zhiyao Luo and Oscar Hou In Chou pointed out in their comparison of SGLT2I versus non-SGLT2I users that the lack of body mass index (BMI) and smoking status can obscure the true relationship between medications and cardiovascular outcomes. This sentiment is echoed by Hao Sen Andrew Fang and Qiao Gao in The effect of oral diabetes medications on glycated haemoglobin (HbA1c) in Asians in primary care, where the inability to account for a patient's diet or exercise makes it difficult to isolate the specific impact of a drug from non-pharmacological influences.

Severity and Systemic Strain

The granularity gap also complicates our understanding of hospital performance. When researchers look at mortality rates in intensive care units, they must distinguish between the quality of care and the "acuity" (the baseline sickness) of the patients.

In a national study of COVID-19 mortality, Harrison Wilde and Thomas A. Mellan found a clear link between ICU occupancy and death in A national retrospective cohort study of mechanical ventilator availability. However, they noted that the absence of granular physiological severity scores at the time of admission makes it difficult to prove that "strain" caused the deaths, rather than the admission of more critically ill individuals during surge periods.

Recovering Detail through Innovation

To overcome these limitations, researchers are moving beyond simple structured data (like ICD-10 codes) and toward more sophisticated extraction methods.

  1. Text Mining: Nicholas J. Leeper and Paea LePendu demonstrated that high-fidelity text-mining of unstructured clinical notes can recover the "missing" details that billing codes ignore, allowing for a more accurate safety profile of drugs like Cilostazol.
  2. Informative Missingness: Rather than treating missing data as a failure, Amelia L.M. Tan and Emily Getzen argue in Informative Missingness: What can we learn from patterns in missing laboratory data? that the absence of a lab test is itself a data point. It reflects a clinician's judgment that a patient is stable, acting as a proxy for severity.
  3. Machine Learning Causal Inference: Modern techniques are being used to "predict" missing variables or better adjust for the ones that do exist. Zhiyao Luo utilized machine learning-based causal inference to provide a "reality check" on drug effectiveness, attempting to simulate the rigors of a clinical trial within the messy landscape of real-world data.

The goal of this research direction is not to replace clinical trials, but to make administrative data "clinical" enough to provide reliable evidence where trials are impossible or unethical. By acknowledging and bridging the granularity gap, researchers can transform massive, noisy datasets into precise instruments for public health.

Papers in this version

Showing 60 of 70 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bridging_the_granularity_gap_in_administ_abf76c89,
  title = {Bridging the Granularity Gap in Administrative Healthcare Research},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-granularity-gap-in-administrative-healthcare-research/rdr_abf76c89101e41249c99b4a946112c8f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_granularity_gap_in_administ_abf76c89_v1,
  title = {Bridging the Granularity Gap in Administrative Healthcare Research},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-granularity-gap-in-administrative-healthcare-research/rdr_abf76c89101e41249c99b4a946112c8f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}