TL;DR
This paper introduces "time-context"—the calendar lookback window—as a critical design axis for Electronic Health Record (EHR) foundation models, distinct from token-context. By evaluating 30 years of primary care data, the authors demonstrate that the predictive signal for colorectal cancer plateaus at approximately 18 years, providing a biological and empirical bound for data curation in clinical AI.
What Problem Does This Paper Address?
Most research into EHR foundation models focuses on "token-context"—the number of medical events a model can process at once. However, in clinical practice, the time elapsed is often more relevant than the number of events. A patient with a chronic condition might generate thousands of tokens in two years, while a healthy patient's entire life might fit into a few hundred.
For slow-progressing diseases like colorectal cancer (CRC), which follows a decades-long "adenoma-carcinoma sequence," it was previously unknown how far back a model must look to capture the earliest predictive signals. Furthermore, most EHR research relies on hospital datasets, which are often "shallow" (covering only a few years), preventing the study of long-range longitudinal signals.
Methods
The authors trained a 34.6M parameter decoder-only transformer on 1.4 billion coded events from 3.6 million primary-care patients.
Key methodological innovations include:
- Nested Evaluation Design: To isolate the effect of time, the authors used a "frozen" cohort of patients who all had at least 30 years of history. They then evaluated the model on the same patients using 15 different lookback windows, ranging from 2 to 30 years.
- Time-Aware Encoding: Each medical event was encoded as a triplet of feature (SNOMED CT or dm+d codes), value, and time (age at event).
- RoPE Retuning: The authors discovered that standard Rotary Position Embeddings (RoPE) "alias" at decade scales—meaning events 30 years apart can appear more similar to the model than events 20 years apart. They retuned the base frequency ($\theta$) to 40,750 to ensure monotonic decay in attention similarity over a 35-year span.
Datasets
The study utilized the Discover dataset, a de-identified research mirror of the Whole Systems Integrated Care (WSIC) database in North West London.
- Scope: ~6M patients and ~2B events from 1990 to 2026.
- Primary Care Focus: Unlike hospital data, this provides a continuous, granular view of patient history over decades.
- Evaluation Cohort: 1,008 CRC cases and 9,582 matched controls, all with $\ge$ 30 years of continuous registration.
Key Insights and Ideas
- The 18-Year Plateau: The predictive signal for CRC doesn't increase indefinitely. It rises steadily until about 18 years and then plateaus. This aligns with the known biological timeline of CRC development (10–26 years).
- Signal vs. Architecture: At deep lookbacks (12+ years), the transformer and a count-based XGBoost baseline performed almost identically. This suggests that for long-range prediction, the signal is a property of the data itself rather than a specific neural architecture.
- Diminishing Returns of Tokens: While the median number of tokens per patient grows significantly as the window expands (from ~90 at 2 years to ~1,200 at 30 years), the marginal tokens added in later years contribute very little to predictive accuracy.
Key Results
- Discrimination Gain: The Transformer's AUROC rose from 0.551 at a 2-year lookback to a peak of 0.590 at 18 years.
- Early Signal Capture: Approximately 80% of the total performance gain was achieved within the first 10 years of history.
- RoPE Aliasing: Simulation showed that at the default $\theta = 10,000$, events 25–35 years apart appeared more similar than events 20 years apart. Retuning $\theta$ restored the expected decay (see Figure 3).
Limitations and Future Work
- Selection Bias: The "deep cohort" (30+ years of history) is naturally biased toward older, residentially stable patients, which may not represent the general population.
- Data Shift: The density and coding quality of EHR data improved significantly from the 1990s to the 2020s, potentially confounding the "depth" signal with "data quality" signal.
- Single Region: The study is limited to North West London.
- Future Directions: The authors suggest expanding this analysis to other cancers (e.g., lung, pancreas) to see if they exhibit different "depth signatures" and scaling the model size to see if absolute discrimination can be improved.
Summary updated