Reinforcement learning for treatment planning breaks down in exactly the cohorts that need it most. On the EpiCare treatment benchmark, offline value-learning methods beat a simple standard-of-care heuristic only once they see more than about 2,000 patient episodes, and below 256 episodes they often do worse than a random policy (EpiCare). Rare-disease registries sit permanently in that second regime. The usual fix is to borrow data from common diseases. Done naively, borrowing hurts: KnowRare found that pooling records from all ICU conditions degrades prediction for the rare ones, and that only carefully selected similar conditions help (KnowRare).
This proposal starts from a property of temporal-difference (TD) learning, the family of methods that update the value of a state from the estimated value of the next state instead of waiting for the final outcome. Because TD builds long-horizon value estimates out of single transitions, it never needs a whole patient trajectory to come from the target disease. It only needs each individual step (state, treatment, next state) to be locally correct. We propose to borrow common-disease transitions one step at a time. Each borrowed transition is weighted by how well it matches the rare disease's own dynamics at that point in state space, and is down-weighted where the two diseases diverge. The claim is that this local, step-level borrowing stops the error compounding that makes small-cohort TD collapse. It also avoids the negative transfer that comes from borrowing whole conditions.
Core Claim
Two results in the file seem to contradict each other, and they motivate the proposal. In tabular Markov chains, least-squares TD estimates rare-event statistics with polynomial sample complexity, $O(n^3)$, while Monte Carlo needs $O(e^n)$, where $n$ is the number of states (TD for rare event prediction). That paper shows TD's accuracy is governed by local state connectivity, not by how rare the event is globally. Yet in the one deep RL study of rare-disease diagnosis, standard TD-based Q-learning was unstable, and the authors had to abandon bootstrapping for Monte Carlo returns computed inside a hand-built expert simulator (Besson et al.).
We read the gap through the result of Fujimoto et al.. When a finite dataset is missing transitions, infinitely many value functions fit the Bellman equation perfectly while carrying arbitrary true error. The critic's error then obeys its own Bellman-like recursion and flows backward through every earlier state (validation buffer paper). In a 100-patient cohort, most successor states are unsupported, so this recursion has nothing to anchor it.
TD's advantage in the tabular rare-event setting comes from dense local coverage. The cure for small cohorts is therefore to restore local coverage, and other diseases can supply it wherever their local dynamics agree with the target's. The prediction that distinguishes this from condition-level transfer is specific. Suppose a source disease matches the rare disease in some regions of state space (for example, early-stage physiology) and differs in others (late-stage decompensation). Then step-level weighted borrowing should beat both rare-only training and whole-condition pooling. When sources match nowhere, it should fall back gracefully to rare-only performance.
Research Program
The method has three parts. The first is condition-level preselection. We rank candidate source conditions by clinical similarity, as KnowRare does with its knowledge graph over diagnosis co-occurrence, variable distributions and shared drugs. This step only narrows the candidate pool. The second is a per-transition weight. For each source transition $(s,a,s')$ we estimate the ratio $\rho(s,a,s') = p_T(s' \mid s,a) / p_S(s' \mid s,a)$ between target and source next-state distributions. Two small classifiers separate target from source data, one on full transitions and one on state-action pairs; their log-odds difference gives the ratio. The third part is reward relabeling. The clinician-defined target reward $r_T$ is recomputed on borrowed transitions, so only dynamics are borrowed, never disease-specific outcomes.
Ratio estimates are unreliable exactly where the rare cohort has no data. In those regions we shrink $\rho$ toward the condition-level prior, using a coverage estimate from the target classifier. The critic then minimizes a weighted TD loss:
$$ L(Q) = \mathbb{E}_{D_T}\big[\delta^2\big] + \lambda\, \mathbb{E}_{D_S}\big[\bar w(s,a,s')\,\delta^2\big], \qquad \delta = Q(s,a) - r_T(s,a,s') - \gamma \max_{a'} \bar Q(s',a'). $$
Here $D_T$ and $D_S$ are the rare and source datasets, $\bar w$ is the clipped, coverage-shrunk ratio, $\lambda$ sets the borrowing strength, $\gamma$ is the discount and $\bar Q$ is a target network. The loss sits on top of conservative Q-learning (CQL), which EpiCare found to be the most reliable method in clinical settings. The weighting mechanism is borrowed from SUNRISE's down-weighted Bellman backups (SUNRISE). The difference is the weighting signal: SUNRISE weights by ensemble disagreement within one environment, while we weight by measured cross-disease dynamics mismatch.
On the analysis side, we will measure true value error directly in simulation. Bellman error can stay low while value error explodes; Fujimoto et al. report value error nearly 57 times the Bellman error for Bellman residual minimization on HalfCheetah. We will also track how error at unsupported successor states propagates backward with and without borrowing. This tests the mechanism, not just the final return.
Baselines
The naive reference points are a random policy and EpiCare's standard-of-care heuristic. Both are evaluated by rollouts in the simulator, so no off-policy estimate is involved. Behavior cloning on the rare cohort alone is the imitation reference.
The direct baseline is CQL trained only on the rare cohort, which EpiCare reports as the strongest offline method overall. The strongest existing data-scarcity method is TreatStitch, which stitches segments of existing patient trajectories, using a Schrödinger bridge (an optimal-transport method for generating smooth connecting paths) where segments do not quite meet (TreatStitch). It reports a cumulative EpiCare score of 40.07 with 1,024 episodes, against 15.79 for CQL alone. We found no reported result on Lacuna for any method below 256 EpiCare episodes. EpiCare states only that performance there "often dropped below random."
Every method gets the same rare cohort, the same source pool, the same network size, and the same hyperparameter search budget. Selection uses cross-fitted held-out rare patients, never simulator returns. Beating TreatStitch means two things. At 1,024 episodes we should exceed 40.07 under its protocol. More importantly, at 32 to 256 episodes we should exceed both the standard of care and TreatStitch when a locally matching source exists.
The transfer baselines are CQL on all source conditions pooled with the target, CQL on KnowRare-style top-$k$ selected conditions pooled with uniform weight, and source pretraining followed by target fine-tuning. The ablations isolate the key ingredient. Uniform weights within selected conditions removes the step-level weighting. An oracle using true ratios from the simulator gives an upper bound on what better weighting could achieve. A no-borrowing variant that truncates bootstrapping at low-coverage successors tests whether gating alone, without borrowed data, explains any gain. A final variant drops the condition-level shrinkage to test the coverage fallback.
Why not just…?
Why not just pretrain the critic on common diseases and fine-tune on the rare cohort? This is the most serious threat. Fine-tuning can adapt to the target without explicit weighting, and with very few target patients it may simply be hard to beat. The weakness we expect is that fine-tuning on 50 patients cannot tell the network which regions to keep and which to overwrite. It inherits source values wholesale in unsupported regions, which is exactly where the target differs. The similarity sweep below settles this. If fine-tuning matches our method at every level of local and global mismatch, the step-level weighting adds nothing, and we will report that.
Why not just pool similar conditions, as KnowRare does? KnowRare's selection operates on whole conditions and was designed for prediction, not bootstrapped values. A condition that is similar on average still imports mismatched transitions in its divergent regions, and TD propagates that bias backward through every earlier state. The uniform-weight ablation compares the two approaches directly on the same selected sources.
Why not just drop bootstrapping and use Monte Carlo returns, or more pessimism? Besson et al.'s Monte Carlo approach required an expert-built simulator to produce complete episodes. Registry trajectories are censored and incomplete. The rare-event analysis also shows Monte Carlo variance growing exponentially where TD grows polynomially. Heavier CQL pessimism is included in the tuning grid, so if it alone suffices, the rare-only CQL baseline will show it. The truncation ablation covers the "just stop bootstrapping into unsupported states" variant.
Why not just augment within the cohort, as TreatStitch does, or shorten the horizon with action chunks? Within-cohort stitching recombines transitions the cohort already contains. It cannot add coverage of states that no rare patient visited, and that is the failure Fujimoto et al. identify. Chunked critics such as Decoupled Q-Chunking shorten error propagation but were validated on datasets of up to 100M transitions, and they still bootstrap from unsupported chunk endpoints. Both approaches are complementary to ours. We will report TreatStitch combined with our weighting to check whether the gains add.
Proposed Experiments
The primary testbed is a family of related simulated diseases built on EpiCare's partially observed progression model. We assume its transition parameters can be modified; if not, we will reimplement a matched model. Each source condition supplies 2,000 or more episodes. The target cohort ranges from 32 to 256 episodes, plus 1,024 episodes for comparison with TreatStitch.
The decisive manipulation is the type of mismatch between source and target dynamics. In the localized condition, sources match the target in some disease stages and differ sharply in others. In the global condition, sources are mildly perturbed everywhere. In the null condition, sources are unrelated to the target. We predict three outcomes. Weighted borrowing should clearly beat pooling and fine-tuning in the localized condition. All transfer methods should be roughly tied in the global condition. Our method should stay near rare-only performance in the null condition, while pooling falls below it.
Metrics are true discounted return and adverse-event rate from online rollouts, true value error at visited and unsupported states, and the gap between our weights and the oracle weights. A pattern where the method helps only through the oracle, or where the truncation-only ablation matches the full method, would refute the borrowing claim.
A secondary check uses rare ICU conditions in MIMIC-III and eICU, with source conditions selected as in KnowRare and fluid and vasopressor actions as in OGSRL. EpiCare reports a direct-method off-policy estimation error (RMSE) of 106.4 on a 100-point return scale, so we will not claim policy improvement from these data. We will instead report held-out next-state likelihood for weighted versus pooled dynamics, and clinician concordance. The simulator results carry the causal conclusion.
References
- EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes
- Bridging data gaps of rare conditions in ICU: a multi-disease adaptation approach for clinical prediction
- The surprising efficiency of temporal difference learning for rare event prediction
- A Model-Based Reinforcement Learning Approach for a Rare Disease Diagnostic Task
- Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
- A Case for Validation Buffer in Pessimistic Actor-Critic
- SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
- Treatment Stitching with Schrödinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategies
- Decoupled Q-Chunking
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies