Reversal-Gated Asymmetric Policy Learning from Small Serial ctDNA Cohorts

Cancer patients on a given therapy are now often monitored with repeated blood draws that measure circulating tumor DNA (ctDNA), fragments of tumor genome shed into the bloodstream. A falling ctDNA level usually means the drug is working. A rising level, or a new mutation appearing, often signals resistance months before a scan shows it. The decision these data should inform is simple to state: keep the current therapy or switch. A model that learns this decision from past patients faces three problems. Cohorts with serial profiling are tiny, often 25 to 115 patients. Most recorded trajectories in refractory disease end badly. And the environment eventually flips, because a therapy that cleared the dominant tumor clone stops working once a resistant clone expands.

We propose to learn the continue-or-switch policy with an asymmetric update. By default, the learner draws mostly on the patients who did well and discounts the many noisy failures, which keeps a small-data policy stable. When the ctDNA trajectory itself shows evidence of clonal escape, the learner turns that discount off, so failures again carry full weight. The two-sentence version is this. Asymmetric learning is a known cure for instability when learning from someone else's mostly failed decisions, but it is also known to make learners blind to reversals. Serial ctDNA gives a measured signal of when a reversal is happening, so the asymmetry can be switched off exactly then.

Core Claim

Two results from different fields point at the same trade-off, and nobody has connected them to a measured biological signal. Asymmetric REINFORCE for off-Policy Reinforcement Learning shows that policy-gradient learning from another policy's samples collapses when the reward baseline sits at or above that policy's value. Lowering the baseline by a conservative shift (around $-0.1$) makes the learner focus on "what to do" from successes and keeps the policy from becoming deterministic too early. That work supplies our method, but it was tested only on bandits and language-model math reasoning. Confirmation bias in human reinforcement learning fits human choices with separate learning rates for good and bad news (about 0.5 versus 0.15). It shows that this positivity bias costs accuracy after reward contingencies reverse. That paper constrains our design: a fixed asymmetry is the wrong answer whenever the world can flip. Tumor evolution under therapy is exactly such a flip, as the lesion-specific resistance clones tracked in plasma by Murtaza et al. illustrate.

Our claim has three parts. First, in offline policy learning from cohorts of fewer than about 100 patients dominated by poor outcomes, a negative baseline shift beats the symmetric update. Second, the best shift shrinks toward zero as the cohort grows, because the variance that the shift suppresses falls with sample size while the bias it introduces does not. Third, a shift gated by a ctDNA-derived escape probability keeps the small-sample benefit without the reversal penalty. Concretely, the policy $\pi_\theta$ is trained on recorded decisions $a_i$ in molecular states $s_i$ with outcomes $R_i$, using

$$ g_\theta = \sum_i \big(R_i - \hat V - \delta_0\,(1 - e(s_i))\big)\,\nabla_\theta \log \pi_\theta(a_i \mid s_i). $$

Here $\hat V$ is the estimated value of the historical clinicians' policy, $\delta_0 > 0$ is the default conservative shift, and $e(s_i) \in [0,1]$ is the estimated probability that a resistant clone is expanding. When $e = 0$ the update is plain AsymRE. When $e = 1$ it is the symmetric update. The term $e(s)$ is computed from the ctDNA history: rising mutant allele frequency after clearance, or a variant absent at the nadir appearing later. What is new is not asymmetry itself but making it depend on a measured signal of reversal.

Research Program

The main obstacle is evaluation. Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination audited offline treatment-policy learning on ICU sepsis data across 17,550 experiments. It found that standard off-policy evaluation estimators often rank random policies above clinicians, and that offline RL often fails to beat imitation of clinicians except in high-risk, rapidly deteriorating subgroups. That paper both motivates our focus on deteriorating, refractory patients and rules out evaluating with importance-sampling estimates on tiny real cohorts. We therefore build a semi-synthetic testbed whose ground truth is known, and use real cohorts only for checks that do not require counterfactual policy values.

The testbed is a mechanistic simulator with five components: sensitive and resistant clones, therapy-specific kill rates, stochastic emergence of resistance, lesion-site-dependent ctDNA shedding, and a clonal hematopoiesis noise floor. Clonal hematopoiesis means mutations in blood from aging non-tumor stem cells; the floor adds non-tumor mutations to plasma. The simulator also models irregular, missing blood draws and radiographic progression as the delayed outcome. We calibrate it to statistics reported in the file. From the colorectal liver-metastasis cohort of Reinert et al., we match 100% recurrence when ctDNA is positive at 30 days versus 55.6% when negative, a median lead time of 2.5 months over CT, and lower shedding from lung than from liver metastases. From the CameL-sq biomarker cohort of Chen et al., we match an objective response rate of 73.8% versus 27.8% by on-treatment blood tumor mutational burden, with baseline values uninformative. Historical behavior policies mimic standard care: continue until radiographic progression, with clinician-like noise.

The simulator supports two "centers" that differ in metastatic site mix, assay noise, and sampling schedule. This lets us test cross-site generalization, the failure mode that small-cohort studies such as Monge et al. and Hoefflin et al. repeatedly report. We draw training cohorts of N = 25, 50, 115 and 270, mirroring the cohort sizes in those papers and in Reinert and CameL-sq. We evaluate every learned policy on 10,000 fresh simulated patients from both centers.

Two pieces are built: the escape-probability estimator $e(s)$ and the gated learner. The estimator is a small Bayesian change-point model over per-variant allele frequencies, with a clonal hematopoiesis noise term; it is fit once and frozen. Because a reviewer will rightly note that $e(s)$ alone could drive a simple switching rule, we evaluate that rule as a baseline in its own right. We also feed $e(s)$ as a state input to every learned policy, so the gate is the only difference between arms.

Baselines

The naive reference is the current standard of care in the simulator: continue the therapy until radiographic progression, then switch. Every arm is reported as a difference from it in simulated progression-free time.

The strongest simple competitor is a ctDNA rule with no learning. One version switches at the first confirmed rise after nadir, which is what the Reinert lead-time result suggests clinicians might adopt. A stronger version switches when the frozen change-point escape probability exceeds a threshold chosen on the training cohort. This baseline is the real threat. It uses the same molecular evidence as our method and has one tunable parameter, so it should resist overfitting well at N = 25.

The imitation baseline is behavior cloning of the simulated clinicians, plus filtered behavior cloning that imitates only trajectories with good outcomes. Filtered cloning is the limit our update approaches as $\delta_0$ grows large.

The strongest existing offline RL methods are Conservative Q-Learning and Implicit Q-Learning as run in the dynamic-treatment-regime audit, along with symmetric off-policy REINFORCE and fixed-shift AsymRE. No reported policy-value result was found on Lacuna for any learned policy on longitudinal ctDNA data. The closest reported numbers are prognostic, not decision-making: the ctDNA-based Bayesian risk score of Fu et al. reports a time-dependent AUC of 0.745 for 3-year progression in its independent test cohort. We reimplement a risk score of that type, thresholded into a switch rule, as an additional baseline.

Every arm gets the same features, including $e(s)$, the same training cohorts and seeds, and the same tuning budget of 30 configurations. Selection uses cross-validated simulator rollouts on the training center only.

The key ablation compares the gated shift against fixed shifts across a sweep of $\delta_0$ values with the gate removed. This isolates the claim that the gate, rather than the asymmetry or the feature, carries the gain. Our method counts as beating the field only under three conditions. It must exceed the tuned change-point switch rule in mean progression-free time on the held-out center, with a confidence interval over 50 cohort resamples excluding zero. It must do so at N ≤ 115. And it must not raise the rate of premature abandonment, meaning switching away from a therapy that was still working.

Why not just…?

Why not just switch on a ctDNA threshold or change-point alarm? Because it may simply be as good. Reinert's 100% positive predictive value suggests that in high-shedding disease a rise really is decisive. Our argument for learning is narrower. The best response to an alarm should depend on context the rule ignores: lesion site and shedding, time on therapy, and alternative options. Where the alarm is unreliable, as with low-shedding lung metastases or clonal hematopoiesis noise, a fixed threshold should either fire late or fire falsely. The low-shedding center tests exactly this. If the rule matches our method there, we will report that learning is unnecessary for this decision.

Why not just use symmetric conservative offline RL such as Conservative Q-Learning? Conservatism keeps the policy near the clinicians' actions, but it still penalizes every observed failure. In refractory cohorts, where the Freiburg tumor-board series saw only 7.1% of all patients respond, those failures dominate. The audit paper found that such methods often fail to beat imitation overall. AsymRE showed that symmetric baselines collapse in off-policy training, though in a very different domain. The head-to-head at N = 25–50 settles whether that transfers.

Why not just filtered behavior cloning? It is the large-shift limit of our update and is cheap. Its weakness is the reversal problem documented in the confirmation-bias study: it learns only from successes, so it never learns to abandon a therapy that used to work. The $\delta_0$ sweep traces the whole curve from filtered cloning to symmetric REINFORCE. If filtered cloning sits at the optimum, the gate adds nothing and we will say so.

Proposed Experiments and Decisive Checks

The first check is the sample-size curve. For each N, we find the best fixed shift and test whether it moves toward zero as N grows, as the variance argument predicts. A flat curve would undercut the mechanism we claim.

The second check is a reversal stress test. We vary the rate and timing of resistance emergence and measure time from resistant-clone expansion to switch. The fixed-shift learner should lag here and the gated learner should not.

The third check is cross-center transfer, under shifts in shedding, assay noise, and sampling irregularity. This is where we expect learned policies most likely to lose to the simple rule.

The fourth check uses real data, and we are explicit about its limits. With CameL-sq-type two-timepoint data we can test only whether the policy's cycle-2 continue or switch recommendation separates outcomes, as on-treatment blood tumor mutational burden does. With the serial cohorts indexed in the BloodPAC Data Commons, we can test whether $e(s)$ alarms precede documented progression. Neither check estimates the counterfactual value of a policy.

A clean negative result would also be informative. If the tuned change-point rule matches every learned policy at all cohort sizes, that would tell the field that the value of serial ctDNA lies in the alarm, not in policy learning.