Machine learning systems can now recommend when each person should return for their next screening scan. A woman judged to be at low risk might come back in three years, and a high-risk one in six months. Before such a schedule reaches clinics, someone has to estimate how well it would have performed on existing patient records. That estimate is badly distorted by how long each patient's record is. Healthy patients contribute long histories with many scans. Patients who develop cancer contribute short histories that stop at diagnosis. The standard reweighting estimators multiply one correction factor per scan decision, so the long healthy histories carry almost no weight, and a handful of short cancer histories end up deciding which schedule looks best. The result is a policy comparison that rests, in practice, on a few individual cases. That is the same problem clinicians warn about when they generalize from single case reports.
We propose an off-policy evaluator for screening schedules built on one structural fact: before a tumor is detected, the choice of when to scan does not change the underlying biology. The decision to scan in month 12 or month 18 can only change the outcome during the preclinical window, the period when a tumor exists but has not yet caused symptoms. Outside that window, the only thing a schedule changes is how many scans the patient receives, and that can be computed by replaying the candidate schedule directly. Our estimator splits a schedule's value into two parts. The first is scan burden, which is replayed exactly across the whole record without any reweighting. The second is early-detection benefit, which is reweighted only across the few decisions that fall inside the preclinical window. Each patient's effective record length therefore shrinks from years of follow-up to the handful of decisions that actually matter. We also report, alongside every policy ranking, how many individual patients that ranking depends on.
Core Claim
Retrospective screening data involve two separate sources of trouble: long, uneven follow-up and a small number of cancer cases. We claim both can be defused by confining importance weights to the preclinical window. An importance weight here is the ratio of how likely the candidate schedule and the historical clinicians were to choose each observed interval. This yields lower error and more stable policy rankings than generic corrections for uneven record length. It also makes the "how many patients decide this" question answerable and small enough to report.
Written out, for patient $i$ with observed follow-up of length $T_i$ ending in clinical diagnosis or censoring, let $W_i$ be the set of decision times inside the estimated preclinical window. Let $a_t$ be the chosen interval and $h_t$ the imaging and risk history at decision $t$. Let $\pi$ be the candidate schedule, $\mu$ the historical scheduling behavior, $G_i$ the early-detection benefit in months, $B_i(\pi)$ the number of scans $\pi$ would order over $T_i$, and $\lambda$ the price of one scan. The estimator is
$$\hat V(\pi) = -\frac{\lambda}{n}\sum_{i=1}^n B_i(\pi) + \frac{\sum_i w_i G_i}{\sum_i w_i}, \qquad w_i = \prod_{t \in W_i} \frac{\pi(a_t \mid h_t)}{\mu(a_t \mid h_t)}.$$
The number of factors in $w_i$ is set by the length of the preclinical window, typically a few scans, and not by $T_i$. That is the source of the variance reduction. It is also why long healthy records stop being ignored: they enter through the exact burden term instead.
Two further design choices matter. First, the evaluation horizon is anchored to each patient's observed clinical timeline, not to the moment the candidate schedule would have detected the tumor. An aggressive schedule that finds a tumor earlier therefore is not penalized for "ending the episode" sooner. This addresses the truncation tension that Metelli et al. raise for finite-horizon evaluation. Second, the start of the window is never observed. We take it from a tumor sojourn-time model and average the estimator over that model's uncertainty, not a single point estimate.
Relation to Prior Work
The motivating evidence comes from length-bias work in reinforcement learning. Importance Sampling for Fair Policy Selection shows that weights on long trajectories vanish. In its Chain MDP benchmark, standard importance sampling picks the correct policy well under half the time as length grows. That paper introduces per-horizon weighted importance sampling (PHWIS), which groups trajectories by length and reweights within each group, and this recovers near-perfect selection. State Relevance for Off-Policy Evaluation (OSIRIS) drops the weight at states where the action does not affect return-to-go, and detects such states with a Welch t-test on discretized states. Our estimator is a domain-specific instance of the OSIRIS idea. The difference is that relevance comes from a known causal fact about screening, not a hypothesis test, so it needs no discretization of volumetric images. That discretization is exactly the scaling barrier the OSIRIS line leaves open.
PHWIS supplies a component we reuse: preclinical windows still differ in length, so we normalize within groups of equal window length. It does not supply the main mechanism. Grouping by total record length does nothing about the scarcity of cancer cases within each group.
The application comes from Optimizing risk-based breast cancer screening policies with reinforcement learning. That paper's system, Tempo, pairs the Mirai image-based risk model with a learned policy that chooses intervals from 6 months to 3 years. It evaluates by replaying schedules against records, using a recurrent network to fill in risk between visits. Tempo is our main competitor for evaluation, and its replay is a model-based estimator whose errors come from the interpolation model. Our proposal does not build a better scheduler. It builds a better way to rank schedulers, and Tempo-style replay becomes one input to it.
HOPE is the closest off-policy evaluation work in healthcare. It infers intermediate rewards and smooths each patient's record by averaging over similar patients, but it evaluates with plain weighted importance sampling and names long-horizon variance as unsolved. We address that open problem directly.
Active Screening for Recurrent Diseases and AdaFuse apply reinforcement learning to screening, but they rely on simulation or on single-timepoint fusion. They do not evaluate schedules offline, so they motivate the setting but do not compete with us.
The single-case literature shapes how results are reported. Case reports such as the esophageal carcinoma response to 5-FU and leucovorin and pancreaticoduodenectomy in a 92-year-old stress that one striking outcome may be an outlier driven by host factors. Lung Cancer Risk Estimation with Incomplete Data shows that screening records are full of skipped visits and missing risk factors. We turn the single-case warning into a measurement. For each ranking, we compute the smallest number of patient records whose removal would reverse it, and we treat a ranking that hinges on one to three patients as unsupported.
Clinical programs such as Ishikawa's hemodialysis renal cancer surveillance motivate adaptive schedules in other high-risk groups. We have no public longitudinal dataset for that setting, so it remains motivation only.
Research Program
The work has three parts. The first is the estimator itself. It consists of the exact burden replay, the within-window weights with per-window-length normalization, and averaging over sojourn-time uncertainty. It also includes a behavior model $\mu$ fitted to observed return intervals, because real adherence varies: in real records the "action" includes a patient returning late or skipping a round.
The burden replay has one assumption we have to test. When a candidate schedule would scan at a time with no observed image, we use the risk score from the nearest real scan. This is justified outside the window, where no tumor is present. Inside the window we never impute; we reweight.
The second part is a semi-synthetic testbed where the true value of every schedule is known. We build it on National Lung Screening Trial (NLST) patients: we keep real risk-factor profiles, sample tumor onset and growth from a sojourn model, and generate records under a stochastic behavior policy that resembles real adherence. This is necessary because NLST's own protocol, three annual screens, is close to deterministic. On the real data alone, importance weights would be nearly degenerate, and no ground truth would exist.
The third part applies the method to real irregular records, NLST rounds with missed visits and the Vanderbilt Lung Screening Program (VLSP) cohort, to see whether the rankings stay stable when we have no ground truth.
As an optional extension, if the evaluator is used to choose between trained schedulers, we would apply the stopping-time gradient correction of Reinforcement Learning with Random Time Horizons during training. We keep this secondary, because the claim of the proposal is about evaluation.
Baselines
The naive reference point is fixed annual screening, both as a policy to be ranked and as the thing any adaptive schedule must beat. Tempo reports 4.10 months of early-detection benefit at 0.96 mammograms per year on the Massachusetts General Hospital (MGH) cohort, against 1.58 months at 1.0 per year for annual screening. On the Karolinska cohort it reports an efficiency of 9.63 months per unit screening frequency, against 6.29. These figures come from the Tempo paper. They show that adaptive schedules can look much better, and that is exactly the kind of claim whose reliability we want to test.
The strongest existing evaluator is Tempo-style replay: a recurrent model interpolates risk between visits, and each candidate schedule is simulated through each record. It is strong because it uses every patient and has no weight variance. Its weakness is bias whenever the interpolation model is wrong. We will run it with the same risk model, the same data splits, and the same hyperparameter search budget as our estimator.
The strongest reweighting competitors are per-decision weighted importance sampling, PHWIS as in the fair policy selection paper, OSIRIS with its t-test run on clustered risk-score states as in State Relevance for Off-Policy Evaluation, and HOPE with neighbor calibration. All of them will share our behavior model and our state features.
For the medical benchmarks, we found no reported result on Lacuna for off-policy evaluation of screening schedules on NLST or VLSP. On synthetic benchmarks, the reported reference points are PHWIS's near-perfect selection accuracy on the Chain MDP and HOPE's Spearman rank correlation of 0.99 on medical data.
Two ablations isolate what is new. The first applies window-restricted weights without the exact burden replay, instead reweighting burden over the full record. This separates the gain from confining the weights from the gain from replaying burden. The second removes per-window-length normalization.
We will count the proposal as beating the baselines if it achieves three things on the semi-synthetic benchmark. It must have lower root-mean-squared error than both replay and PHWIS. It must have higher rank correlation with the true policy values. And its median count of decisive patients must be clearly larger, showing that its rankings rest on many records rather than a few. In addition, its error must not blow up when the sojourn model is misspecified.
Why not just…?
Why not just use Tempo-style replay, since screening does not change biology anyway? This is the most serious alternative, and it may win whenever the interpolation model is accurate. Its failure mode lies inside the preclinical window. There, filling in risk between scans means guessing how visible a tumor would have been on an image nobody took, which is the quantity that decides the detection benefit. Our semi-synthetic benchmark lets us control how wrong the interpolation is. If replay matches our estimator even under heavy misspecification, the honest conclusion is that confining the weights to the window adds nothing. In that case we would report the window decomposition only as a variance diagnostic.
Why not just use PHWIS? PHWIS fixes vanishing weights by grouping records of equal length. In screening, record length mostly reflects enrollment date and censoring, and each group contains only a few cancers. The weighting over the long disease-free portion is still there, and so is the dependence on a few decisive cases. The decisive-patient count across PHWIS groups will show whether this concern holds in practice.
Why not just rank patients with static risk groups and assign fixed intervals? This improves the schedule, not the evaluation. Static risk-stratified schedules are simply one more policy to be ranked, and Tempo's results suggest they leave detection benefit unrealized. We include them in the comparison set, so if the evaluator says they are as good as adaptive schedules, that finding is itself informative.
Why not just run OSIRIS with its statistical test? The t-test needs discrete states and a significance threshold. With few cancers, it will often fail to detect that decisions inside the window are relevant, or it will flag irrelevant decisions by chance. Comparing its relevance labels against the known windows in the semi-synthetic benchmark measures this directly.
Proposed Experiments
We will start with sanity checks on the Chain MDP and the Dilly-Dallying Gridworld, modified so that the steps where the action matters are known in advance. There, window-restricted weighting should reproduce PHWIS-level selection accuracy with lower error.
The main experiment is the NLST-based semi-synthetic benchmark. We will vary cancer prevalence, window length, how noisy adherence is, and how badly the sojourn model is misspecified. For each estimator we will report root-mean-squared error, Spearman correlation with true policy values, accuracy in picking the best policy, and the decisive-patient count.
The decisive test comes in two parts. As prevalence falls toward realistic levels, the weighted-importance-sampling baselines, PHWIS included, should show decisive-patient counts collapsing toward single digits while our estimator's counts stay substantially higher. If they do not, the central claim fails. Under misspecification, we will map where replay's bias overtakes our estimator's variance.
On the real NLST and VLSP irregular records, we will check that rankings stay stable across resampled patient sets and across plausible sojourn models. We will also report the decisive-patient count for every published-style claim that an adaptive schedule beats annual screening.
References
- A Tale of Sampling and Estimation in Discounted Reinforcement Learning
- Importance Sampling for Fair Policy Selection
- State Relevance for Off-Policy Evaluation
- Optimizing risk-based breast cancer screening policies with reinforcement learning
- HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcare
- Active Screening for Recurrent Diseases: A Reinforcement Learning Approach
- AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learning
- A RESECTED CASE OF ADVANCED ESOPHAGEAL CARCINOMA SHOWING A PROMINENT RESPONSE TO NEOADJUVANT CHEMOTHERAPY WITH 5-FU AND LEUCOVORIN
- Laparoscopic pancreaticoduodenectomy in A-92-older Chinese patient for cancer of head of the pancreas
- Lung Cancer Risk Estimation with Incomplete Data: A Joint Missing Imputation Perspective
- Renal cell carcinoma in chronic hemodialysis patients in Japan
- Reinforcement Learning with Random Time Horizons