This paper, recently published by researchers at Tel Aviv University, addresses a major inefficiency in how we analyze Electronic Health Record (EHR) data. By rethinking how we handle "prevalent" cases—patients who already have a condition when they join a study—the authors have unlocked a way to nearly double the discovery power of genetic association studies.
TL;DR
Standard survival analysis often discards "prevalent" data (events occurring before study recruitment) due to computational complexity, leading to a massive loss of statistical power. This paper introduces a pairwise pseudolikelihood approach that incorporates this retrospective information without the need for unstable numerical integration, doubling the number of genetic discoveries in a real-world bladder cancer study.
What Problem Does This Paper Address?
Large-scale biobanks like the UK Biobank (UKB) are "delayed entry" systems. Participants are recruited at various ages (e.g., between 40 and 69) and then followed prospectively. However, many participants have already experienced the "event" of interest—such as a cancer diagnosis—years before they were ever recruited. These are called prevalent cases.
The standard statistical tool, the Cox proportional hazards model, typically handles this via "left truncation." It essentially ignores everything that happened to a person before they walked through the recruitment center doors. While this avoids certain biases, it is incredibly wasteful. In the case of Urinary Bladder Cancer (UBC) in the UKB, prevalent cases make up about 40% of all observed events. Throwing them away is like trying to solve a puzzle with nearly half the pieces missing.
Existing methods to include this data require "conditional likelihood" approaches that involve complex numerical integration for every single participant during every step of the optimization. For a dataset like the UKB with 500,000 people, this is computationally prohibitive and prone to "crashing" (numerical instability).
Methods
The authors propose a Pairwise Pseudolikelihood Approach. Instead of trying to calculate a likelihood for the entire population at once (which creates the nasty integration problem), they look at pairs of observations.
- The "Cancellation" Trick: By conditioning on the order statistics of a pair of individuals, the mathematically difficult parts of the likelihood function (the denominators requiring integration) simply cancel out.
- Illness-Death Model: They embed the disease onset within a three-state stochastic model (Healthy $\rightarrow$ Diseased $\rightarrow$ Death). This allows them to account for the fact that to be a "prevalent" case, you had to survive long enough with the disease to be recruited.
- Subsampling for Speed: Calculating all $n^2$ possible pairs in a large biobank is impossible. The authors show that a subsampling technique—selecting a fixed number of pairs ($K_n$) per observation—maintains statistical consistency and efficiency while keeping the computation time manageable ($O(n)$ complexity).
- Nuisance Parameters: They use a "plug-in" method where parameters for death and censoring are estimated first using standard methods and then plugged into the pairwise formula to solve for the primary coefficients of interest (e.g., genetic risk factors).
Datasets
- Simulated Data: The authors ran extensive simulations across three settings (A, B, and C) to test the method against different levels of covariate correlation and model misspecification.
- UK Biobank (UKB): A massive repository of ~500,000 UK individuals.
- Urinary Bladder Cancer (UBC) Study: They focused on 31 Single Nucleotide Polymorphisms (SNPs) previously identified in Genome-Wide Association Studies (GWAS) to test the "replicability" of these genetic markers using their new method.
Key Insights and Ideas
- Waste Not, Want Not: The "aha moment" is realizing that prevalent data is only "unusable" because of our current computational tools, not because the data itself is inherently flawed for time-fixed variables like genetics.
- Robustness to Misspecification: Surprisingly, the method remains highly effective even if the secondary models (like the model for when a patient might die of other causes) are slightly wrong.
- The Power of Pairs: Pairwise pseudolikelihood acts as a "proxy" for the full conditional likelihood. It provides the efficiency gains of the complex model with the stability of the simpler one.
Key Results
- Efficiency Gains: In simulations, the proposed method showed a Relative Efficiency (RE) of 1.3x to 2.1x compared to the standard approach. This means you would need a much larger sample size using standard methods to get the same precision.
- Real-World Discovery: In the UK Biobank UBC replication study:
- The standard method detected only 6 significant genetic associations (SNPs).
- The proposed pairwise method detected 11 significant associations.
- Computational Feasibility: Using $K_n = 50$ pairs per observation was sufficient to achieve these results, proving the method scales well to "Big Data" EHR contexts.
Limitations and Future Work
- Recall Bias: The authors acknowledge that prevalent data relies on retrospective reporting. While this is fine for genetic data (which doesn't change), it is risky for time-dependent lifestyle factors (like "how much did you exercise 10 years ago?") due to memory errors.
- Time-Dependent Covariates: The current pairwise framework struggles with covariates that change over time because the "swapping" of outcomes in the pairwise logic creates "quasi-observations" with incomplete histories.
- Model Expansion: Future work could extend this approach to other survival models, such as Accelerated Failure Time (AFT) models, or incorporate "penalized" likelihoods (like Lasso) for high-dimensional data.
Reference: Keret, N., & Gorfine, M. (2023). Unlocking Retrospective Prevalent Information in EHRs - a Pairwise Pseudolikelihood Approach. arXiv:2309.01128v1.
Summary updated