A needle biopsy samples a few cubic millimetres of a tumor that may hold several genetically distinct cell populations. When a resistant or drug-sensitive subpopulation sits outside the needle's path, the biopsy reports the wrong biomarker status, and the patient gets a treatment their tumor did not call for. Oncology treats this as pure noise. This proposal treats it as something offline reinforcement learning badly needs: variation in treatment that does not depend on outcome. Offline reinforcement learning means learning a treatment policy from historical records without running new trials. Clinicians act deterministically on the biopsy readout, but the readout is a spatially random function of the true tumor. So, measured against the true tumor state, their historical choices are stochastic, and the probability of each choice can be computed from tumor geometry.
The central claim is concrete. Suppose the learner describes each patient by whole-tumor imaging features that capture how much of the tumor carries the relevant subclone, rather than by the biopsy label. Then the historical data covers the best imaging-based policy with a coverage constant we can write in closed form from the subclone volume fraction and the number of biopsy cores. Pessimistic offline methods can then provably, not just empirically, improve on biopsy-driven care. The same analysis predicts an uncomfortable trade-off: image-guided biopsies that aim at suspicious regions reduce sampling bias but also destroy this randomization, and with it the ability to learn from the resulting records.
Core Claim
Offline RL theory no longer requires data that visits every state-action pair. It needs only single-policy concentrability: the historical data must place non-negligible probability on the states and actions that one good policy would visit. This is summarized by $C^* = \sup_{s,a} d^{\pi^*}(s,a)/\mu_B(s,a)$, where $d^{\pi^*}$ is the visitation distribution of the target policy and $\mu_B$ is the data distribution. Pessimistic algorithms achieve error governed by $C^*$ alone. Examples are pessimistic value iteration in Is Pessimism Provably Efficient for Offline RL?, lower-confidence-bound Q-learning in Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity, and the augmented-Lagrangian method ALMIS in Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian. All of these assume the logged state is the true state.
Clinical logs violate that assumption in a particular way. The clinician's choice depends on the biopsy label $b$, while outcomes depend on the true subclone fraction $f$. In the simplest one-decision case, cores are placed at random and are small relative to the scale of heterogeneity. A clinician then gives targeted therapy whenever any of $k$ cores detects the subclone, which happens with probability $p(f) = 1-(1-f)^k$.
Let the target policy give targeted therapy exactly when $f \ge f^*$. Its coverage constant with respect to the biopsy-driven behavior is
$$C^*(f^*, k) = \max\left\{ \frac{1}{1-(1-f^*)^k},\ \frac{1}{(1-f^*)^k} \right\},$$
because the worst-covered patients are those just on either side of the threshold. $C^*$ is finite for any interior threshold, even though the clinician never randomized.
Spatial clustering of subclones replaces $f$ with the needle's hit probability given lesion geometry. Repeated re-biopsy across a treatment course multiplies such factors, so coverage degrades with horizon at a rate the geometry determines. Two things are new here: the source of coverage, and the fact that it can be calculated.
The claim holds only if the imaging features actually carry the information in $f$. If they carry only part of it, the biopsy label becomes a hidden confounder. It is correlated with the action and also with the outcome through the part of $f$ that imaging misses. The guarantee then fails. We measure that failure directly rather than assume it away.
Research Program
The first component is a simulator. DTR-Bench (DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime) supplies chemotherapy (AhnChemoEnv) and mixed radio-chemotherapy (GhaffariCancerEnv) models, but they track a single lumped tumor population. They corrupt observations only with Gaussian noise and random dropout, which the file's gap analysis correctly notes cannot represent spatially correlated sampling bias. We extend both environments with a three-dimensional lesion made of sensitive and resistant subpopulations. Each subpopulation grows under the existing ordinary-differential-equation dynamics and occupies spatially clustered regions with a tunable correlation length.
Observations come in two forms. The first is a needle biopsy: a line of voxels along a sampled trajectory, whose placement can be random or aimed at the most "suspicious" imaging region. The second is a whole-volume imaging summary built from histogram, texture, and habitat features. It follows the handcrafted-feature tradition of Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach and the K-means habitat partition of Habitat-based radiomics analysis for evaluating immediate response in colorectal cancer lung metastases treated by radiofrequency ablation. An information knob controls how much of the subclone fraction survives in this summary, emulating the resolution floor of imaging.
Behavior policies imitate clinical practice. They choose therapy deterministically from the biopsy label and re-biopsy at progression. The simulator therefore provides exact $d^{\pi^*}$ and $\mu_B$, so $C^*$ can be computed both in true-state space and in imaging-feature space.
The second component is theory. We prove the one-step coverage formula above and its spatially clustered version. We then extend it to $H$-step regimes with re-biopsy and derive the resulting suboptimality bound for a linear pessimistic learner. The whole-tumor imaging features serve as the feature map of a linear Markov decision process (a model in which rewards and transitions are linear in known features), which is the object that A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs and Offline Primal-Dual Reinforcement Learning for Linear MDPs take as given. Those papers supply the coverage notion in feature space. We supply a mechanism that produces coverage and a formula for it.
We also quantify the confounding penalty. When imaging retains only a fraction of the information about $f$, how much bias enters the pessimistic estimate? A sensitivity bound in the style of marginal sensitivity models is the intended form.
The third component is the learner. We run pessimistic value iteration with an elliptical uncertainty penalty on the imaging features, which is PEVI as in Jin et al. We also run the In-Sample Actor-Critic of The In-Sample Softmax for Offline Reinforcement Learning, because it never queries treatments absent from the data. One variant uses the known geometric propensity $p(f)$ in place of an estimated behavior policy. This addresses the file's noted open problem that in-sample and primal-dual methods need a behavior density they usually cannot estimate well. Here geometry supplies that density.
Baselines
The naive reference point is the historical biopsy-driven policy itself, evaluated in the simulator. Behavior cloning of that policy on imaging features is included too. Buckman et al. (The Importance of Pessimism in Fixed-Dataset Policy Optimization) show that imitation is capped at demonstrator performance, and this pair measures how much headroom exists at all.
The strongest non-RL competitor is a supervised virtual biopsy followed by a treatment rule. Concretely, a network predicts biomarker status from imaging, as the PET/CT model of Non-invasive decision support for NSCLC treatment using PET/CT radiomics does. That model reports an EGFR-status AUC between 0.81 and 0.86 across its cohorts. The predicted status then drives treatment. We train it once on biopsy labels, which is realistic, and once on oracle labels, which is an upper bound. Beating it means higher simulated progression-free time than both versions at equal data.
The strongest offline-RL competitor is Conservative Q-Learning (CQL), the standard empirical pessimistic method. On real data, the file reports CQL reaching a cumulative reward of 15.79 against 40.07 for TreatStitch in a restricted MIMIC-III sepsis setting (Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategies). On MIMIC-III, the guarded method GMB-CPO reports a mortality estimate of 0.0138 against a standard-of-care 0.0632, with concordance 0.549 (Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies). These are sepsis numbers and do not transfer to our environments. No reported result was found on Lacuna for any offline RL method on a spatially heterogeneous tumor environment, and the file gives no exact DTR-Bench reward values.
All methods share the same logged datasets, the same imaging features, and the same network sizes. Each gets an equal hyperparameter budget, selected by a fixed offline criterion; in the simulator we additionally report oracle-selected results to separate algorithm quality from tuning luck.
The ablation that isolates the key ingredient runs the identical pessimistic learner under four state representations: the biopsy label, imaging features with degraded subclone information, full imaging features, and the oracle subclone fraction. The claim predicts that performance tracks the computed $C^*$ and the retained information, not the algorithm.
Why not just…?
Why not just train a supervised model to predict the biomarker from imaging and treat by rule? Trained on biopsy labels, such a model learns the biopsy's errors. This is the ground-truth disconnect the file flags. Even trained on perfect labels, it answers "is the subclone present?" rather than "at what subclone burden does targeted therapy pay off, and when should we switch?" In a single decision with a known optimal threshold, a well-calibrated predictor could genuinely match our method. That is a real threat, and the one-step experiment is designed to show it if true. The expected advantage lies in multi-step regimes where the threshold depends on outcomes and on re-biopsy timing.
Why not just regress outcome on imaging and action, one model per treatment? In a single-decision setting, this contextual-bandit approach with a lower confidence bound is essentially what pessimism reduces to. We include it and expect parity at horizon one. The proposal's value then rests on two things: the coverage analysis, which tells us when that regression is trustworthy, and the sequential setting, where per-step regression ignores downstream consequences.
Why not just apply CQL or another standard regularizer without any coverage analysis? CQL may perform comparably on some settings. The point of the analysis is to predict in advance, from geometry, when any pessimistic learner can succeed. That includes the targeted-biopsy regime, where we predict failure regardless of algorithm. Rashidinejad et al. also show that behavior-regularization penalties carry an asymptotic bias that keeps them off the $1/\sqrt{N}$ rate, which our sample-size sweeps can check.
Why not just take more biopsy cores? Repeated needle passes carry real morbidity, and in deep thoracic or brain lesions they are often not feasible. The formula shows that increasing $k$ reduces coverage for patients with small subclone fractions, so more cores trade diagnostic accuracy against learnability of the records. We sweep $k$ to make that trade explicit.
Proposed Experiments
The first decisive check validates the formula. Across subclone correlation lengths, core counts, and threshold policies, the simulator-computed $C^*$ in imaging space should match the closed form, and the learned policy's suboptimality should scale with $\sqrt{C^*/N}$ as the pessimism bounds predict. If suboptimality does not track $C^*$, the central claim fails.
The second check is the randomization paradox. Switching from random to image-targeted needle placement should improve biopsy accuracy while driving $C^*$ toward infinity and pessimistic-policy value down toward behavior cloning.
The third check measures the confounding dose-response. As the imaging summary loses subclone information, the gap over the biopsy policy should shrink. It should eventually reverse for unpenalized methods, and our sensitivity bound should predict where.
The fourth check compares horizons: parity with outcome regression at one step, and a growing advantage with re-biopsy over three to six decisions.
As a real-data sanity check, we would apply the coverage estimator to retrospective PET/CT cohorts with TKI and ICI arms of the kind in the NSCLC decision-support study, if access can be obtained. There we would report estimated $C^*$ rather than policy value, since counterfactual outcomes are unavailable.
References
- Is Pessimism Provably Efficient for Offline RL?
- Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
- Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
- DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime
- Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach
- Habitat-based radiomics analysis for evaluating immediate response in colorectal cancer lung metastases treated by radiofrequency ablation
- A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs
- Offline Primal-Dual Reinforcement Learning for Linear MDPs
- The In-Sample Softmax for Offline Reinforcement Learning
- The Importance of Pessimism in Fixed-Dataset Policy Optimization
- Non-invasive decision support for NSCLC treatment using PET/CT radiomics
- Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategies
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies