How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model

Achille Raviez, Lan Wang, Brendan DelaneyarXiv 2026 · arXiv

TL;DR

This paper introduces "time-context"—the calendar lookback window—as a critical design axis for Electronic Health Record (EHR) foundation models, distinct from token-context. By evaluating 30 years of primary care data, the authors demonstrate that the predictive signal for colorectal cancer plateaus at approximately 18 years, providing a biological and empirical bound for data curation in clinical AI.

What Problem Does This Paper Address?

Most research into EHR foundation models focuses on "token-context"—the number of medical events a model can process at once. However, in clinical practice, the time elapsed is often more relevant than the number of events. A patient with a chronic condition might generate thousands of tokens in two years, while a healthy patient's entire life might fit into a few hundred.

For slow-progressing diseases like colorectal cancer (CRC), which follows a decades-long "adenoma-carcinoma sequence," it was previously unknown how far back a model must look to capture the earliest predictive signals. Furthermore, most EHR research relies on hospital datasets, which are often "shallow" (covering only a few years), preventing the study of long-range longitudinal signals.

Methods

The authors trained a 34.6M parameter decoder-only transformer on 1.4 billion coded events from 3.6 million primary-care patients.

Key methodological innovations include:

  • Nested Evaluation Design: To isolate the effect of time, the authors used a "frozen" cohort of patients who all had at least 30 years of history. They then evaluated the model on the same patients using 15 different lookback windows, ranging from 2 to 30 years.
  • Time-Aware Encoding: Each medical event was encoded as a triplet of feature (SNOMED CT or dm+d codes), value, and time (age at event).
  • RoPE Retuning: The authors discovered that standard Rotary Position Embeddings (RoPE) "alias" at decade scales—meaning events 30 years apart can appear more similar to the model than events 20 years apart. They retuned the base frequency ($\theta$) to 40,750 to ensure monotonic decay in attention similarity over a 35-year span.
The nested design used to evaluate the same cohort across different time windows.

Datasets

The study utilized the Discover dataset, a de-identified research mirror of the Whole Systems Integrated Care (WSIC) database in North West London.

  • Scope: ~6M patients and ~2B events from 1990 to 2026.
  • Primary Care Focus: Unlike hospital data, this provides a continuous, granular view of patient history over decades.
  • Evaluation Cohort: 1,008 CRC cases and 9,582 matched controls, all with $\ge$ 30 years of continuous registration.

Key Insights and Ideas

  • The 18-Year Plateau: The predictive signal for CRC doesn't increase indefinitely. It rises steadily until about 18 years and then plateaus. This aligns with the known biological timeline of CRC development (10–26 years).
  • Signal vs. Architecture: At deep lookbacks (12+ years), the transformer and a count-based XGBoost baseline performed almost identically. This suggests that for long-range prediction, the signal is a property of the data itself rather than a specific neural architecture.
  • Diminishing Returns of Tokens: While the median number of tokens per patient grows significantly as the window expands (from ~90 at 2 years to ~1,200 at 30 years), the marginal tokens added in later years contribute very little to predictive accuracy.
AUROC performance for the Transformer and XGBoost across different lookback windows.

Key Results

  • Discrimination Gain: The Transformer's AUROC rose from 0.551 at a 2-year lookback to a peak of 0.590 at 18 years.
  • Early Signal Capture: Approximately 80% of the total performance gain was achieved within the first 10 years of history.
  • RoPE Aliasing: Simulation showed that at the default $\theta = 10,000$, events 25–35 years apart appeared more similar than events 20 years apart. Retuning $\theta$ restored the expected decay (see Figure 3).
Simulation of RoPE attention similarity over 35 years for different base frequencies.

Limitations and Future Work

  • Selection Bias: The "deep cohort" (30+ years of history) is naturally biased toward older, residentially stable patients, which may not represent the general population.
  • Data Shift: The density and coding quality of EHR data improved significantly from the 1990s to the 2020s, potentially confounding the "depth" signal with "data quality" signal.
  • Single Region: The study is limited to North West London.
  • Future Directions: The authors suggest expanding this analysis to other cancers (e.g., lung, pancreas) to see if they exhibit different "depth signatures" and scaling the model size to see if absolute discrimination can be improved.

Summary updated

Abstract

EHR foundation models read medical events within a fixed context window, and long context has so far been measured only in tokens. How far back in time a model must look to capture the earliest predictive signal is unknown. We introduce time-context, the calendar lookback window before the prediction point, and provide, to our knowledge, the first systematic evaluation of decade-scale time-context for an EHR foundation model. We pretrained a decoder-only transformer (34.6M parameters) on 1.4B coded events from 3.6M primary-care patients in North West London and fine-tuned it to predict incident colorectal cancer under a 12-month pre-diagnostic buffer. A frozen cohort of 1,008 cases and 9,582 matched controls, all registered for at least 30 years, was evaluated at 15 nested lookback windows from 2 to 30 years, so that every truncation is compared within the same patients. Discrimination rose from an AUROC of 0.551 at 2 years to 0.590 at 18 years, then plateaued, with about 80% of the gain reached within 10 years; an XGBoost baseline converged to the same level (peak AUROC 0.589). A simulation further suggests that rotary position embeddings (RoPE) over elapsed time alias beyond about 20 years at the default base frequency, motivating a retuned $θ$. Time-context is a measurable design axis for clinical prediction models, distinct from token-context, and should be reported.
BibTeX
@misc{raviez2026how,
  author = {Achille Raviez and Lan Wang and Brendan Delaney},
  title = {{How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model}},
  year = {2026},
  eprint = {2611.07265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07265},
}
Figure