Crediting the Priming Dose: A Transcriptomic Testbed for Temporal Credit Assignment

When a cancer drug sequence fails or succeeds, clinicians usually cannot tell which decision in the sequence mattered. A tumor treated with drug A for months, switched to drug B, then drug C, may collapse at step three because of a collateral vulnerability that A created weeks earlier — or despite it. Today's computational pharmacogenomics cannot even pose this question: models like Precily and oncoPredict map a single snapshot of tumor RNA expression to an immediate sensitivity score, and the field's own systematic assessment confirms that essentially all of these predictors are static, one-step regressions from baseline expression to a 72-hour viability readout (Systematic Assessment of Analytical Methods for Drug Sensitivity Prediction from Cancer Cell Line Data).

Meanwhile, the reinforcement-learning literature has spent a decade building exactly the missing machinery: temporal credit assignment — methods that attribute a delayed outcome back to the specific earlier actions that caused it, rather than to whatever happened right before the reward. These methods (hindsight importance weighting, transformer-based return decomposition, non-Markovian reward modeling) excel on locomotion tasks with delayed rewards, but have never touched a real biological decision problem; the field's own survey flags the absence of non-synthetic benchmarks as a core limitation (A Survey of Temporal Credit Assignment in Deep Reinforcement Learning). This proposal builds the bridge: a sequential drug-scheduling environment whose state is the tumor's transcriptome, learned from real perturbation data, and a study of whether delayed-reward attribution methods can both find better drug sequences and correctly name the priming dose.

Core Claim

The claim has two parts, one methodological and one scientific. Methodologically: a usable sequential-therapy environment can be built by learning a one-step perturbation-response model from harmonized single-cell perturbation data (scPerturb: Harmonized Single-Cell Perturbation Data) and composing it across treatment cycles, with a static sensitivity predictor (Gene expression based inference of cancer drug sensitivity) supplying the drug-response readout. Scientifically: on this environment, credit assignment methods trained only from a terminal outcome — no intermediate rewards — will both (a) find sequencing policies that beat myopic per-cycle re-screening when the dynamics contain hysteresis, and (b) recover which earlier intervention opened a collateral-sensitivity window, which no static predictor can express even in principle.

This is deliberately not "RL for drug design." PaccMannRL already does RL conditioned on a transcriptomic vector, but as a single-step bandit: it generates a molecule, scores it immediately with a static predictor, and never models a tumor that evolves in response to treatment. Playing the Long Game: Using Stackelberg Mean-Field Games to Outsmart Cancer is the closest prior work on the therapy side — it optimizes multi-cycle dosing against delayed resistance emergence and shows that anticipating long-horizon evolution beats greedy protocols — but its tumor is a three-population differential-equation abstraction with no genome-wide state, no drug-specific chemistry, and no credit assignment algorithm; its evaluation is 20 synthetic virtual patients. Our proposal differs on both axes at once: high-dimensional transcriptomic state grounded in real perturbation data, and explicit delayed-reward attribution algorithms whose per-action credits can be scored against ground truth.

Research Program

The build has three components. First, the one-step dynamics model: train a conditional generative model on scPerturb's 44 harmonized datasets (over 2.5 million cells) to map (current pathway-level state, drug identity, dose, exposure duration) to a distribution over next states. We follow scPerturb's own subsampling guideline — at least 300 cells per perturbation and 1,000 UMI counts per cell — because below that threshold perturbation shifts cannot be statistically separated from noise (scPerturb: Harmonized Single-Cell Perturbation Data). State representation uses pathway activity scores rather than raw genes, the same dimensionality-reduction move that makes Precily trainable (Gene expression based inference of cancer drug sensitivity) and that the systematic assessment identifies as necessary given ~20,000 genes against ~1,000 cell lines (Systematic Assessment of Analytical Methods for Drug Sensitivity Prediction from Cancer Cell Line Data). Model quality is checked by whether predicted state-shift distributions match held-out real perturbations under the E-distance statistics scPerturb uses.

Second, the response head: a Precily-style predictor mapping (pathway state, SMILES fingerprint) to predicted $\log(IC_{50})$, trained on GDSC/CCLE and validated on the independent CTRPv2 split, where Precily reports a Pearson correlation of $\rho = 0.84$ (Gene expression based inference of cancer drug sensitivity); we treat approaching that number as the acceptance bar for the head. Precily's reported zero-shot transfer from bulk to single-cell expression ($\rho = 0.85$) is our evidence that a response model trained on bulk screens can be queried on perturbed single-cell-derived states without collapsing.

Third, the environment and agents. Composing the one-step model over 4–12 treatment cycles gives an environment where the action is (drug, dose) from a restricted panel with dense scPerturb coverage, and the reward is terminal only: a composite outcome such as resistance dominance (state drift beyond a threshold E-distance from the drug-sensitive manifold) together with a cumulative-dose penalty. Terminal-only reward is the realistic clinical setting — longitudinal biopsies are rare, so clinicians effectively observe outcomes, not processes — and Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective proves this is not a fatal handicap: outcome supervision matches process supervision up to a polynomial ($H^3$) horizon factor, provided credit is imputed through the Advantage function rather than Q-values. That result directly shapes our ground-truth definition for attribution. Alongside the data-grounded environment we build injected-hysteresis variants in which a specific priming drug at a specific cycle deterministically opens or closes a later vulnerability window; these variants have known causal structure, letting us score attribution accuracy — something no existing biological or locomotion benchmark provides, since exact information-theoretic attribution (MALLOC) is intractable at scale (On Temporal Credit Assignment and Data-Efficient Reinforcement Learning).

The agents under test reuse existing credit assignment machinery rather than inventing new algorithms. Hindsight credit assignment reweights each past action by the ratio of the hindsight action distribution to the policy,

$$\frac{h(a_t \mid s_t,\, \text{outcome})}{\pi(a_t \mid s_t)},$$

which converts delayed attribution into a supervised learning problem and stays robust to intermediate noise that breaks standard Monte Carlo methods (Hindsight Credit Assignment). Transformer return decomposition distributes an episodic return across timesteps with bias-corrected policy gradients (Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning). CoDeTr handles the case where the terminal outcome is a non-linear composite of history — exactly the shape of a resistance threshold — using bidirectional in-sequence attention rather than the sum-of-rewards assumption (Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning). CoDeTr's relaxation of the Markovian assumption matters here because tumor response is the canonical hysteresis case: response at cycle $t$ depends on the whole exposure history, not just the current state.

A known risk, flagged by the survey itself, is that all of these methods inherit the calibration of their surrogate models — a miscalibrated hindsight or return regressor injects systematic bias into policy updates (A Survey of Temporal Credit Assignment in Deep Reinforcement Learning). We therefore include calibration diagnostics (reliability of $h$ against realized actions on held-out trajectories) as a first-class measurement, not an afterthought.

Proposed Experiments

Experiment 1 validates the composed environment. The one-step dynamics model is evaluated on held-out perturbation pairs by E-distance between predicted and real post-perturbation distributions; the response head is evaluated on CTRPv2 against the reported $\rho = 0.84$ benchmark. We additionally report a divergence diagnostic for multi-cycle rollouts — whether simulated states at cycle $k$ stay within the empirical state manifold — since composing single-step models off-distribution is the weakest link in the design and must be quantified, not assumed away.

Experiment 2 compares policies on both the data-grounded environment and the injected-hysteresis variants. Metrics are terminal outcome (time-to-resistance-dominance, in cycles), cumulative dose, and regret against the known optimal policy on injected variants. The comparison set is specified in Baselines below. The key scientific question is honest and open: does the gap between credit-assigned policies and myopic re-screening appear only when hysteresis is injected, or does the data-grounded environment already contain enough history-dependence to produce it?

Experiment 3 scores attribution. On injected variants with a known priming drug at cycle 1–3 of a 4–12 cycle trajectory, we measure top-1 attribution accuracy (does the method name the right cycle and drug) and rank correlation between assigned credits and the ground-truth advantage, following the theoretical result that Advantage, not Q, is the correct process-level proxy (Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective). Recency-based credit (uniform or exponentially decaying with recency) is the attribution baseline; the survey's core observation is that standard methods are exactly recency heuristics in disguise (A Survey of Temporal Credit Assignment in Deep Reinforcement Learning).

Baselines

The do-nothing reference is continuous maximum-tolerated-dose monotherapy: always administer, at full dose, the drug with the lowest predicted $IC_{50}$ against the baseline expression profile. This mirrors the MTD arm in the Stackelberg mean-field-game study, where greedy maximal dosing produced the fastest competitive release of resistant clones — resistance dominance at 180 days versus over 500 days for the long-horizon adaptive policy, which also used 50% of the cumulative dose (Playing the Long Game: Using Stackelberg Mean-Field Games to Outsmart Cancer). It anchors what happens when nobody thinks about delay at all.

The strongest competitor, and the one a clinician would actually recognize, is myopic per-cycle re-screening: at every cycle, re-sequence (here, re-derive) the current transcriptomic state, run a static sensitivity predictor on it, and give the best-scoring drug. We run two instantiations so the comparison is fair to both the simple and the strong end of static prediction: Elastic Net / Ridge regression, which the systematic assessment of over 110,000 models found to be the best performance-interpretability tradeoff, with continuous targets beating binarized classification at $P < 2.2 \times 10^{-16}$ (Systematic Assessment of Analytical Methods for Drug Sensitivity Prediction from Cancer Cell Line Data), and a Precily-style deep network over pathway scores and SMILES embeddings (Gene expression based inference of cancer drug sensitivity). Fairness means identical state encoders, identical drug panels, and an equal tuning budget (the same random-search allocation per method) so a win cannot be attributed to extra compute. Beating this baseline means a statistically significant improvement in terminal outcome at equal or lower cumulative dose across held-out drugs, on both environment families.

The strongest existing sequential method is standard PPO with a discounted reward, which is what a competent RL practitioner would deploy before reaching for specialized credit assignment. Its published weakness on delayed tasks is severe — on episodic Walker2d, standard PPO scored 266 where transformer return decomposition scored 3217 (Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning) — but we retrain it on our environment rather than assuming that transfer. We also port the mean-field-game adaptive controller as a mechanistic baseline (Playing the Long Game: Using Stackelberg Mean-Field Games to Outsmart Cancer): it optimizes long-horizon dosing but with no transcriptomic state, so it tests whether genome-wide state actually buys scheduling value or whether three population counts suffice.

The key ablation removes only the credit assignment ingredient while keeping everything else: the same policy network and training budget, but with credits replaced by a uniform or recency-weighted distribution over the trajectory (vanilla policy gradient over the episode), and, separately, CoDeTr with the sum-of-rewards assumption restored (Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning). If the full methods do not beat these ablations on attribution accuracy and terminal outcome, the claimed ingredient is not doing the work.

Why not just…?

Why not just re-screen with a static predictor at every progression? This is the most serious alternative and we treat it as the primary baseline rather than arguing around it. The mechanistic evidence says greedy per-state selection is evolutionarily blind: in the Stackelberg study, heuristic adaptive cycling delayed resistance only to 310 days versus over 500 for the long-horizon policy (Playing the Long Game: Using Stackelberg Mean-Field Games to Outsmart Cancer), and Precily's own xenograft tracking shows pathway dependencies shifting under drug pressure in ways a single snapshot misreads (Gene expression based inference of cancer drug sensitivity). But that evidence comes from a three-state ODE world, not from data. Our experiment 2 settles it directly: if myopic re-screening matches credit-assigned policies on the data-grounded environment, we will report that the transcriptomic dynamics as currently measurable contain insufficient hysteresis for sequencing to matter — itself a useful negative result — and the injected-hysteresis variants then quantify exactly how much history-dependence is needed before the gap appears.

Why not just standard PPO with discounting? Discounting handles time only by shrinking it; it does not tell the gradient which action earned the reward. The published gap on delayed benchmarks is an order of magnitude (266 versus 3217 on episodic Walker2d, Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning), and standard algorithms degrade sharply when the terminal outcome is a non-linear composite of history at delays of 100–500 steps (Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning). We are honest that our horizons are short in depth (4–12 cycles, not 500 steps); our difficulty is concentrated in density (terminal-only reward) and breadth (branching over drug-dose combinations), the other two axes in the survey's taxonomy. That is precisely why PPO is run head-to-head rather than dismissed: at this depth the gap may shrink, and the experiment is designed to detect whether it does.

Why not just low-dimensional mechanistic ODE models? Classical mathematical oncology already optimizes sequential dosing with a handful of interpretable parameters, and the mean-field-game work shows it works (Playing the Long Game: Using Stackelberg Mean-Field Games to Outsmart Cancer). What those models cannot represent is drug-specific structure: which compound, out of a panel, opens a collateral vulnerability in which pathway state. The systematic assessment shows gene expression carries most of the explainable variance in drug-specific response across hundreds of compounds (Systematic Assessment of Analytical Methods for Drug Sensitivity Prediction from Cancer Cell Line Data), and scPerturb shows single-cell drug responses are high-dimensional distributional shifts that need E-statistics, not scalar population counts, to even measure (scPerturb: Harmonized Single-Cell Perturbation Data). Still, this is a real threat to the scheduling half of the claim, which is why we port the mechanistic controller into our environment as a baseline: if it matches the transcriptomic policies on outcome, then the genome-wide state is unnecessary for scheduling (though still necessary for attribution, which the ODE model does not attempt), and we would say so.

Decisive Checks

Three results decide the proposal. The environment passes only if the response head approaches the reported cross-dataset correlation ($\rho = 0.84$ on CTRPv2, Gene expression based inference of cancer drug sensitivity) and the one-step dynamics model reproduces held-out perturbation distributions under E-distance; otherwise no downstream claim is meaningful. The policy claim stands only if at least one credit assignment method beats myopic re-screening and PPO on terminal outcome at matched tuning budget, and beats its own uniform-credit ablation. The attribution claim stands only if top-1 identification of the priming dose exceeds the recency baseline at priming depth of two or more cycles on injected variants. If the first holds but the latter two fail, the finding is that currently measurable transcriptomic perturbation dynamics are too memoryless for delayed-reward structure to matter — a concrete, publishable negative result that would redirect both fields: it would tell the credit assignment community that its next benchmark needs richer biological hysteresis, and the pharmacogenomics community that static re-screening is, for now, enough.