Modern cell therapies are finally titratable. A doxycycline-inducible CAR against CD147 can be switched on and off on a 24–48 hour timescale Doxycycline Inducible Chimeric Antigen Receptor T Cells Targeting CD147 for Hepatocellular Carcinoma Therapy, and synthetic agonistic receptor (SAR) T cells kill only when a soluble bispecific antibody bridges them to the tumor, so effector activity follows drug concentration minute by minute Bispecific antibodies redirect synthetic agonistic receptor modified T cells against melanoma. Yet the dosing schedules for these systems are still picked by hand, while the failure they must manage — on-target off-tumor toxicity, where the therapeutic kills healthy tissue that shares the target antigen Hurdles of CAR-T cell-based cancer immunotherapy directed against solid tumors — is precisely the kind of rare, irreversible event you cannot learn about by exploring in patients.
This is, structurally, an offline reinforcement learning problem: learn a dosing policy from fixed logs of prior cohorts, never probing the unexplored dose–timing combinations. And offline reinforcement learning has a documented failure mode that maps onto this clinical hazard with uncomfortable precision. When a learned value function is queried on state–action pairs absent from the data, the Bellman $\max$ operator selectively amplifies positive approximation errors until values diverge Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. The safe-offline-RL benchmark literature adds the twist that matters most for toxicity: in constrained settings, the cost function fails by systematic underestimation, so hazardous actions are misclassified as safe Datasets and Benchmarks for Offline Safe Reinforcement Learning.
Core Claim
My claim is that the two extrapolation errors need opposite corrections, and that no existing method applies them in the right directions for this domain. Let $Q_r(s,a)$ be the learned value of tumor reduction (reward) and $Q_c(s,a)$ the learned value of cumulative off-tumor cytolysis (cost), where the state $s$ carries tumor burden, antigen-density profile, and organ-function markers, and the action $a$ is a dosing decision. The pessimism literature pushes reward values down in poorly supported regions — pessimistic Q-learning achieves this with minimax-optimal sample complexity Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity, and variants penalize values via learned dynamics Conservative State Value Estimation for Offline Reinforcement Learning, predictive uncertainty Value Penalized Q-Learning for Recommender Systems, or buffer-anchoring thresholds Frictional Q-Learning. But penalizing a cost function downward makes the agent more reckless, not less. The correct asymmetric update is:
$$Y_r = r - \alpha\,u_r(s,a) + \gamma \max_{a'}\big[Q_r(s',a') - \alpha\,u_r(s',a')\big]$$
$$Y_c = c + \beta\,u_c(s,a) + \gamma \max_{a'}\big[Q_c(s',a') + \beta\,u_c(s',a')\big]$$
where $u_r, u_c$ are support-based uncertainty measures (ensemble disagreement or inverse visitation counts) and $\alpha, \beta$ shrink toward zero as support grows. The sign flip on the cost target is the whole point: an untested dose–timing combination is treated as carrying the worst cost observed anywhere in the data until experience says otherwise — "toxic until proven safe." This is the computational dual of the biological logic gates that require positive evidence of safety before killing Synthetic biology approaches for improving the specificity and efficacy of cancer immunotherapy, such as inhibitory CARs that veto cytotoxicity when a healthy-tissue antigen is present.
This also separates the proposal from the closest existing work. C2IQL avoids extrapolation entirely through expectile regression over only dataset-supported actions, adds a cost reconstruction model to undo the discounting trap, and conditions the policy on a remaining safety budget C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning. I reuse its cost reconstruction and budget conditioning, but its strict in-sample stance cannot propose any dose schedule outside the logged support at all — the over-conservatism ceiling that the stitching and model-based literature identifies as its own open problem Sample Efficient Offline RL via T-Symmetry Enforced Latent State-Stitching, Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism. Asymmetric conservatism licenses measured extrapolation on the reward side while refusing it on the cost side.
Research Program
Three deliverables. First, an offline benchmark — call it OffTumorDose — built from published dynamics rather than invented ones. The SAR–bispecific environment uses the single-cell RNA-seq melanoma dataset GSE72056 (~4,000 cells, 19 patients, MCSP and TYRP1 density distributions) to model heterogeneous antigen expression, and its complete tumor clearance in 40% of mice as the efficacy endpoint Bispecific antibodies redirect synthetic agonistic receptor modified T cells against melanoma; the inducible-CAR environment encodes the measured Tet-On 3G kinetics (24-hour expression peak, return to baseline within 48 hours of doxycycline withdrawal) with off-tumor cost driven by basal CD147 on healthy tissue Doxycycline Inducible Chimeric Antigen Receptor T Cells Targeting CD147 for Hepatocellular Carcinoma Therapy; a third environment models myelosuppression from targeting antigens shared with hematopoietic stem cells, the canonical AML hazard CD37 is a safe chimeric antigen receptor target to treat acute myeloid leukemia. Tumor dynamics extend the ODE tumor-growth-inhibition structure already used for RL dosing studies Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection. Offline datasets are generated by logging behavior policies clinicians actually resemble — maximum-tolerated-dose continuous schedules, fixed-interval schedules, reactive de-escalation — with coverage deliberately varied so some datasets never contain the intermittent high-dose region a better policy would want to visit. Second, the algorithm: asymmetric conservative Q-learning with undiscounted cumulative organ-damage budgets reconstructed via C2IQL's approach, and a policy $\pi(a|s,\hat{L})$ that becomes more cautious as a patient's remaining organ reserve $\hat{L}$ depletes. Third, the diagnostic science: measuring where and by how much plain offline Q-learning underestimates $Q_c$ as a function of dataset support, which no robotics benchmark has connected to a domain where the cost is irreversible.
Baselines
The naive reference point is standard fixed scheduling at maximum tolerated dose, the current clinical default; the action-derived-reward study quantifies its trade-off directly — its human expert policy achieved 61.04% tumor reduction but only by sustaining continuous maximum drug concentration, i.e., maximum cumulative toxicity, while an adaptive agent achieved 35.97% reduction while skipping entire cycles Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection. Beating the fixed schedule means matching or exceeding its tumor reduction while holding cumulative off-tumor cost under a certified budget, at per-patient adaptivity the fixed schedule cannot have. The strongest existing method is CIQL from the C2IQL paper, re-run on our datasets — expectile in-sample learning, cost reconstruction, and budget conditioning, which on DSRL outperformed BCQ-Lag and BEAR-Lag, methods that "often violated safety constraints," and stayed below the safety limit more consistently than the Constrained Decision Transformer C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning; no reported result found on Lacuna for CIQL's numerical scores on individual DSRL tasks, so we will establish its numbers on our environments ourselves under identical data, compute, and tuning budgets. The Constrained Decision Transformer is included as the sequence-modeling contender, since the DSRL benchmark study found it superior to value-based methods precisely because it replaces value bootstrapping with return conditioning Constrained Decision Transformer for Offline Safe Reinforcement Learning. The key ablation swaps our upward cost inflation for symmetric pessimism that lowers both $Q_r$ and $Q_c$; if the sign flip does not matter, constraint-violation rates should be indistinguishable, and the proposal's central claim fails. A second ablation removes the cost reconstruction model, which C2IQL's own ablations showed degrades performance via the discounting trap.
Why not just…?
Why not just use fixed clinical schedules? This is the incumbent, and the file's evidence against it is quantitative: roughly two-thirds of Phase 2 and 3 oncology trials fail on dosing balance, expert continuous-maximum dosing buys tumor reduction at maximum toxicity, and individualized policies beat one-size-fits-all trial schedules Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection. Fixed schedules also cannot respond to the between- and within-patient antigen-density heterogeneity documented in GSE72056 Bispecific antibodies redirect synthetic agonistic receptor modified T cells against melanoma. Our fixed-schedule baseline will make this concrete on the benchmark rather than assumed.
Why not just clone the safe patients? Behavioral cloning on filtered "safe" cohorts is the simplest way to dodge value estimation, and the DSRL benchmark study already tested it: cloning all data violates safety constraints, cloning only safe trajectories yields an overly timid policy that fails the primary objective, and both are misled by lucky trajectories where low toxicity occurred by chance Datasets and Benchmarks for Offline Safe Reinforcement Learning. We will reproduce this failure on OffTumorDose; it is a genuine risk only if it turns out therapeutic objectives are achievable within the safe-cloned support, which the coverage-narrowing experiments will directly measure.
Why not just subtract a toxicity penalty from the reward? The scalarized objective $R = R_{\text{tumor}} - \lambda \cdot \text{Toxicity}$ is what the chemotherapy-dosing work actually used, and its own results expose the weakness: the manual penalty multiplier trivially flips the agent between expert-like aggressive dosing (penalty 0) and ultra-sparse regimens, with no guarantee that cumulative organ damage stays below any certified limit Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection. Lagrangian offline methods built on this idea, BCQ-Lag and BEAR-Lag, repeatedly violated constraints, and a single $\lambda$ cannot adapt to patients with different organ reserves — exactly what budget-conditioned policies were shown to handle across tightened thresholds C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning. We include BCQ-Lag as a baseline rather than argue from citation alone.
Why not just train online in the simulator, DTR-Bench style? This is the real threat, since we are building simulators anyway. The honest answer is that online RL is the right answer for a known simulator and the wrong answer for the deployment problem: the scientific question is what a policy can safely extract from static logs of past cohorts whose dangerous dosing regions were never explored — which is why extrapolation error exists at all — and DTR-Bench itself shows online algorithms degrade under pharmacokinetic variance, observation noise, missing values, and partial observability DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime. We will run online TD3 and SAC under those same perturbations; if online training on the unperturbed simulator matches our safety record under all perturbation levels, the offline formulation earns its keep only in the narrow-coverage regimes, and we will say so plainly.
Proposed Experiments
The decisive experiments are four. First, the cost-underestimation diagnostic: on held-out out-of-support dose–tissue pairs with simulator ground truth, measure signed error of $Q_c$ for plain offline Q-learning versus our inflated $\overline{Q}_c$; the claim predicts plain Q-learning underestimates cost increasingly as support shrinks, while $\overline{Q}_c$ stays above true cost (a recall-style guarantee on toxicity). Second, coverage sweeps: shrink dataset support systematically and track constraint-violation rate and tumor reduction; plain Q-learning should diverge first, symmetric pessimism next, asymmetric conservatism last. Third, budget conditioning: vary the organ-reserve limit across small, middle, and large values and verify the conditioned policy adapts where unconditioned baselines fail, mirroring the threshold analysis in C2IQL. Fourth, robustness: apply DTR-Bench's perturbation protocol (PK/PD variance, noise, missing values) and check the safety guarantee survives model mismatch, the regime where SimGlucose-style environments saw "no algorithm achieving positive rewards" DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime. Success is tumor reduction within ten percent of the best unsafe method at a constraint-violation rate near zero on narrow-coverage datasets. As an outlook, the same asymmetric principle applies to offline receptor design, where the existing TCRPPO approach optimizes binding and sequence naturalness but no off-tumor cross-reactivity term at all T-Cell Receptor Optimization with Reinforcement Learning and Mutation Policies for Precesion Immunotherapy — but no cross-reactivity dataset exists in the present file, so that extension stays out of scope until data supports it.
References
- Doxycycline Inducible Chimeric Antigen Receptor T Cells Targeting CD147 for Hepatocellular Carcinoma Therapy
- Bispecific antibodies redirect synthetic agonistic receptor modified T cells against melanoma
- Hurdles of CAR-T cell-based cancer immunotherapy directed against solid tumors
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- Datasets and Benchmarks for Offline Safe Reinforcement Learning
- Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
- Conservative State Value Estimation for Offline Reinforcement Learning
- Value Penalized Q-Learning for Recommender Systems
- Frictional Q-Learning
- Synthetic biology approaches for improving the specificity and efficacy of cancer immunotherapy
- C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning
- Sample Efficient Offline RL via T-Symmetry Enforced Latent State-Stitching
- Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
- CD37 is a safe chimeric antigen receptor target to treat acute myeloid leukemia
- Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection
- Constrained Decision Transformer for Offline Safe Reinforcement Learning
- DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime
- T-Cell Receptor Optimization with Reinforcement Learning and Mutation Polices for Precision Immunotherapy