Muscle-Aware Offline Reinforcement Learning for Chemotherapy Dose Modification

Oncologists routinely lower, delay, or hold chemotherapy doses when patients develop toxicity. Standard doses are scaled by body surface area, a height-and-weight formula that cannot tell muscle from fat. Yet routine staging CT scans already measure a patient's skeletal muscle, and low muscle mass predicts complications, toxicity-related harm, and death across cancers and in the ICU. We propose to learn dose-modification policies from historical treatment records using offline reinforcement learning, meaning a decision rule trained only from logged data, with no trial-and-error on patients. The state of each patient would include CT-measured muscle, updated at every restaging scan. We then ask a narrower question than "does adding imaging help": does tracking muscle loss between scans change which dose decisions are best, beyond what a single baseline muscle measurement already gives?

That question matters because the cheap version of this idea may be enough. If a one-time rule such as "start low-muscle patients at a reduced dose" captures most of the benefit, sequential decision machinery adds nothing. The proposal is built so that this outcome is detectable and is reported as a result, not hidden.

Core Claim

Body composition behaves unusually as a state variable in sequential decision-making. It is measured rarely, only when a CT happens to be ordered. The timing of scans is itself informative, since sicker patients are imaged more often. It is measured with pipeline noise, and it changes partly because of the treatment being chosen. Our claim is that an offline RL agent which keeps a filtered estimate of "muscle reserve" will choose measurably better dose modifications than two alternatives: an agent blind to body composition, and an agent given only baseline muscle. The filtered estimate is anchored by CT measurements and propagated between scans using weight, albumin, and cycle-level labs. The advantage should be concentrated in patients with accelerated muscle loss during therapy. The claim is falsifiable at three levels. First, muscle must improve prediction of next-cycle toxicity. Second, the dynamic estimate must beat the static one. Third, both must beat a supervised toxicity model with a threshold rule.

Why this gap is real

On the clinical side, the evidence motivates the state design but stops at prognosis. Meta-analyses find CT-measured sarcopenia predicts ICU mortality (Sarcopenia as a predictor of mortality among the critically ill in an intensive care unit: a systematic review and meta-analysis). Gastric cancer cohorts go further and show that muscle loss after surgery, not only before, worsens survival (Preoperative sarcopenia and postoperative accelerated muscle loss negatively impact survival after resection of locally advanced gastric cancer). Low muscle mass and low muscle radiodensity are associated with complications after robotic gastrectomy (Association of low skeletal muscle mass and radiodensity with clinical outcomes in patients undergoing robotic radical gastric cancer surgery: a population-based retrospective cohort study). Radiodensity is how bright the muscle appears on CT, and it falls as fat infiltrates the muscle. Finally, patients with normal BMI can hide severe muscle depletion (Factors associated with muscle mass in community-dwelling older people in Singapore: Findings from the SHIELD study). All of these studies treat muscle as a static risk marker. None uses it to choose actions over time. The accelerated-loss finding is the specific motivation for a sequential, rather than one-shot, formulation.

On the RL side, two dosing papers supply methods but are blind to host tissue. Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection trains a double Q-network inside a simulated glioma tumor-growth model. Its reward is tumor shrinkage minus a penalty on dose. It learns online in the simulator, with no patient-specific host biology. We reuse its simulator as the skeleton of a semi-synthetic testbed but add a muscle compartment. Realistic CDSS Drug Dosing with End-to-end Recurrent Q-learning for Dual Vasopressor Control is the closest methodological template. It trains Conservative Q-Learning (CQL) on MIMIC-IV and eICU records. CQL pushes down value estimates for actions the clinicians rarely took. The paper also uses stepwise "nudge" actions and recurrent replay, and it contains no imaging features. We adopt its action structure and recurrent critic essentially unchanged, so any gain we report is attributable to the muscle state rather than to a new RL algorithm.

Three further works constrain the design. The imaging RL that does exist, Deep Reinforcement Learning for L3 Slice Localization in Sarcopenia Assessment, uses a Q-network only to find the L3 vertebra slice. It is a perception tool and says nothing about treatment. We therefore do not use RL for perception at all. Constrained Offline Policy Optimization (COPO) gives confidence bounds on a policy's cost from logged data. We use it to cap the predicted rate of severe toxicity rather than tuning a toxicity penalty weight by hand. Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning shows that separating variables the actions cannot affect lets one simulate unobserved actions. Muscle is not such a variable here, because chemotherapy drives muscle loss. We therefore use that paper's idea only partially: skeletal frame, height, sex, and age are treated as exogenous, while muscle is treated as endogenous and slow.

Research Program

Perception layer. The perception layer is deliberately standard. Following Fully-Automated Analysis of Body Composition from CT in Cancer Patients Using Convolutional Neural Networks, a regression network locates L3 and a 2D U-Net segments muscle, subcutaneous fat, and visceral fat. That paper reports Dice 0.95–0.98, R=0.99 against manual areas, and a 9.4 mm mean slice error. Dice measures pixel overlap between predicted and manual segmentations. From each scan we compute the skeletal muscle index (muscle area divided by height squared) and mean muscle radiodensity. A 9.4 mm slice error is not negligible for area, so we estimate a per-scan measurement variance by re-segmenting at neighbouring slices. This variance is passed to the filter rather than discarded.

Muscle-reserve filter. Let $m_t$ be latent muscle index at treatment cycle $t$, $y_t$ a CT measurement available only on scan cycles, and $x_t$ cycle-level labs and weight. We fit a simple state-space model:

$$m_{t+1} = m_t + f_\theta(m_t, a_t, x_t) + \epsilon_t, \qquad y_t = m_t + \eta_t,$$

Here $a_t$ is the dose action, $f_\theta$ is a small learned drift, $\epsilon_t$ is process noise, and $\eta_t$ is measurement noise whose variance comes from the perception layer. The critic receives the filtered mean, its variance, and time since the last scan. This is the key ingredient. The alternative, carrying the last CT value forward, silently treats a three-month-old measurement as current.

Decision problem. One decision is made per chemotherapy cycle. Actions are the clinically operable set: maintain, reduce one dose level, delay, or hold, in the style of the stepwise nudges of Zou et al. The reward combines relative dose intensity delivered and progression-free status at restaging. Severe (grade 3–4) toxicity and unplanned admission are handled as a COPO-constrained cost with a fixed budget, not as a penalty term. Learning uses recurrent CQL with double Q-learning, matched to the vasopressor paper.

Handling informative scanning. Scan timing is informative, so we include the scan indicator itself as an observed input. We also report results separately for patients scanned on a protocol schedule, where timing is not driven by clinical deterioration.

Data

The main cohort must be one that pairs cycle-level chemotherapy records with serial CT. The file identifies no public dataset that does this. MIMIC-IV and eICU, as used by Zou et al., provide ICU time series, and the file contains no evidence that they include linked abdominal CT. The Cancer Imaging Archive ( Integration of Radiomic and Multi-omic Analyses Predicts Survival of Newly Diagnosed IDH1 Wild-Type Glioblastoma) supplies longitudinal oncology CT with outcomes, but no reported dosing-log benchmark was found. We therefore plan three data sources. The primary cohort is an institutional gastric or colorectal cancer cohort with restaging CT every two to three cycles. The gastric setting is chosen because of the accelerated-muscle-loss evidence. TCIA provides external validation of the perception layer and of the muscle-trajectory model. A semi-synthetic benchmark provides ground truth. It extends the Yauney–Shah tumor-growth simulator with a muscle compartment in which drug exposure per unit muscle drives toxicity and dosing drives muscle loss. Its parameters are fitted to the real cohort's observed trajectories. The semi-synthetic benchmark is essential, because on real logs we can only estimate policy value, and the vasopressor paper itself notes that weighted importance sampling is high-variance.

Baselines

The naive reference is the logged clinician policy. Its value is computed exactly on the semi-synthetic data and estimated on real data. A second naive reference is the fixed protocol: body-surface-area dosing with guideline dose reductions triggered by toxicity grade. Every learned policy must beat these on value at equal or lower severe-toxicity rate.

The strongest existing method is recurrent CQL with stepwise actions from Realistic CDSS Drug Dosing with End-to-end Recurrent Q-learning for Dual Vasopressor Control, using all clinical features but no body composition. On vasopressor dosing it reports "more than a 3x improvement in expected reward compared to standard continuous-dosing baselines" under weighted importance sampling. That is a different task and a different comparator, so it serves as evidence the recipe works, not as a target number. We found no reported result on Lacuna for offline RL on chemotherapy dose modification with or without body composition. We will run this baseline with identical splits, network size, and hyperparameter search budget, differing only in whether muscle inputs are present. Beating it means a higher true value on the semi-synthetic data at the same toxicity budget. On real data it means a higher fitted-Q-evaluation estimate whose bootstrap interval excludes the baseline.

Two ablations isolate the key ingredient. Static-muscle CQL receives only the baseline scan. Carry-forward CQL receives the last scan value without filtering. The central claim requires the filtered agent to beat both, and specifically in the subgroup whose muscle index falls by more than 5% between scans. The threshold will be fixed before looking at outcomes.

A supervised competitor is a gradient-boosted next-cycle toxicity model given the same features, including filtered muscle. It is turned into a policy by reducing the dose whenever predicted severe toxicity exceeds a threshold, with the threshold tuned on validation data. Behavioral cloning with muscle inputs completes the set.

Why not just…?

Why not just adjust the starting dose by lean mass instead of body surface area? This is the most serious threat. If most of the benefit comes from correcting the first dose, a static rule suffices. The static-muscle ablation and a lean-mass starting-dose rule applied to the logged cohort will measure this directly. If they match the filtered agent, we will report that the sequential formulation is unnecessary. The prior reason to expect otherwise is the gastric cancer evidence that loss during treatment independently harms survival (Preoperative sarcopenia and postoperative accelerated muscle loss negatively impact survival after resection of locally advanced gastric cancer). A starting-dose rule cannot respond to that loss.

Why not just predict toxicity with a supervised model and threshold it? For single-cycle decisions this may be as good as RL, and we include it as a full baseline. It ignores that holding a dose now changes the muscle trajectory and the options at the next cycle. The semi-synthetic benchmark contains exactly this delayed effect, so the gap between the threshold rule and CQL there measures how much planning matters. If the gap vanishes on real data, the honest conclusion is that the clinical signal is mostly one-step.

Why not just clone what clinicians did, conditioned on muscle? Behavioral cloning reproduces clinicians' habits, including the rigid maximum-dose schedules criticized in Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection. D4RL shows cloning falls behind when good behavior must be stitched together from mixed-quality data (D4RL: Datasets for Deep Data-Driven Reinforcement Learning). Still, if clinicians already reduce doses for visibly wasted patients, cloning may do well. It is included under identical inputs, and its gap to CQL on the semi-synthetic data will be reported.

Proposed Experiments and Decisive Checks

The first check is a cheap go/no-go gate run before any RL. On the real cohort, does filtered muscle index improve held-out prediction of next-cycle severe toxicity? We compare against body surface area, weight, and labs by likelihood and AUROC. If it does not, the Markov-violation argument fails for this population and we stop.

The second check is the semi-synthetic comparison. It includes a negative control: a version of the simulator in which muscle has no effect on toxicity. In that version, all muscle-aware agents must tie the blind baseline. A gain there would expose leakage or tuning artifacts. In the muscle-dependent version, we report true policy value, severe-toxicity rate against the COPO budget, and the error of each off-policy estimator relative to ground truth. The estimator errors tell us how far to trust the real-data estimates.

The third check is on real data. We report fitted-Q and weighted-importance-sampling estimates with bootstrap intervals, alongside a concordance analysis: do patients whose actual care happened to match the learned policy have fewer severe toxicities, after adjustment? This analysis is confounded, and we present it as supporting evidence, not proof. Throughout, results are reported separately for the accelerated-loss subgroup and for protocol-scheduled scans. A win that appears only where scans were triggered by deterioration would suggest the agent learned to read scan timing, not muscle.