Learning Kidney-Sparing Cuts from Slow, Late Feedback

When a surgeon removes a kidney tumor but keeps the rest of the kidney (a partial nephrectomy), the goal is to cut out every bit of tumor while sacrificing as little healthy kidney tissue as possible. Healthy tissue that is cut away cannot be replaced, and losing enough of it pushes patients toward dialysis. This matters most for patients who start with weak kidneys or have tumors on both sides (scoping review on synchronous bilateral renal tumors; case of a patient with chronic renal failure who avoided dialysis through partial nephrectomy). A robot that learned to plan such cuts would have to learn from a simulator that captures how the kidney deforms when it is grasped, pulled, and cut. Simulators accurate enough to judge whether a cut left tumor behind are slow, and their run time varies widely from one case to another. Simulators fast enough for reinforcement learning (learning by trial and error at the scale of millions of attempts) are too crude to judge margins.

This note proposes to split the learning system so that many fast "actors" practice cuts on a crude, quick deformation model, while a separate pool of slow, accurate evaluators re-scores a selected subset of those cuts with a finite-element tissue model. Their verdicts on margins and lost healthy tissue arrive late and are fed back as a learned correction to the crude model's safety cost. The new piece is not distributed reinforcement learning applied to surgery. It is a way to train a policy under a safety constraint when the trustworthy constraint signal is expensive, sparse, and delayed relative to the policy it is judging. Kidney-sparing tumor excision is a natural first test of that problem.

Core Claim

Architectures that separate acting from learning were built to keep learners busy while slow or uneven environments run elsewhere. Examples include SURREAL, with its actors, buffer, learner, and parameter server; AREAL, which tolerates variable-length rollouts; and NitroBox, which addresses latency skew across workers. Online Intrinsic Rewards from LLM Feedback applies the same separation to a slow reward model, pulling it out of the fast training loop.

We claim that the same separation, aimed at a slow high-fidelity safety evaluator rather than a slow environment, lets a policy reach the accuracy of the high-fidelity model at close to the throughput of the crude one. The condition is that delayed evaluator labels are corrected for the gap between the policy that produced a trajectory and the policy now being trained. A second claim can be tested on its own: the places where the crude and accurate tissue models disagree about margins cluster in a small set of situations, such as heavy retraction or cutting near the hilum. A residual cost model can therefore learn the disagreement from a few thousand accurate evaluations instead of millions.

Formally, let $V(\pi)$ be the expected healthy kidney volume remaining after the excision executed by policy $\pi$. Let $C(\pi)$ be the probability that any tumor voxel remains, the positive-margin rate, as judged by the accurate model. Let $T$ be the warm-ischemia step budget: the number of actions allowed while the renal artery is clamped, mirroring the clinical 20–30 minute limit. We seek

$$\max_\pi V(\pi) \quad \text{s.t.} \quad C(\pi) \le \delta, \;\; \text{episode length} \le T,$$

where $\delta$ is a tolerated margin-violation rate. The learner never sees the accurate per-step cost $c_{\text{hi}}$ directly. It sees the crude cost $c_{\text{lo}}$ plus a residual model $r_\phi(s,a) \approx c_{\text{hi}}(s,a) - c_{\text{lo}}(s,a)$, which is trained on labels that arrive $d$ learner updates after the trajectory was generated. Each label is weighted by a clipped ratio $\min(\bar\rho, \pi_{\text{now}}(a|s)/\pi_{\text{gen}}(a|s))$, where $\pi_{\text{gen}}$ is the policy that generated the trajectory, $\pi_{\text{now}}$ is the current policy, and $\bar\rho$ is a clipping threshold. This borrows the clipping idea behind V-trace and the delay-aware correction in Retroactive Advantage Correction. The constraint is enforced with a standard Lagrange multiplier.

Research Program

We would build the task, the two simulators, the split training system, and the analysis of how delay affects the safety constraint.

The task. Kidney and tumor geometry come from the expert masks in the KiTS19 contrast-CT dataset (KiTS19 challenge). Each case is meshed into a volumetric kidney with an embedded tumor. A da Vinci–style instrument arm, with kinematics following SurRoL and TissueRetract, retracts with one tool and cuts with the other. The policy outputs incremental cutting-tool motions and retraction forces. Episodes end at tumor removal or at the ischemia budget.

The two simulators. The crude model is a position-based deformable model, vectorized on GPU in the spirit of Surgical Gym, which is itself rigid-only. The accurate model is a nonlinear finite-element solver with element removal for cutting. It runs on CPU workers, and its per-trajectory cost varies with how many elements are cut.

The evaluator stream. Evaluators replay a sampled trajectory's actions in the accurate model. They return the per-step margin cost, the final remaining volume, and the per-step deformation disagreement. Trajectories are sent for evaluation preferentially where the residual model's ensemble is uncertain.

The delay analysis. We measure how constraint satisfaction degrades as evaluator latency grows, with and without the ratio correction. SURREAL reports that contact-rich tasks "remained unsolved even with 64 actors." That result constrains the design: we do not rely on actor count, and we seed exploration with scripted geometric demonstrations, as SurRoL's HER+DEMO and the Cycle-of-Learning results in TissueRetract suggest is necessary.

We also include one robustness check that comes from the kidney side. KiTS19 tumor masks are less reliable than kidney masks: the winning entry reached tumor Dice 0.851 against kidney Dice 0.974. We therefore evaluate margins against tumor masks dilated and perturbed to mimic under-segmentation. This tests whether policies hedge sensibly when tumor extent is uncertain.

Baselines

Open-loop geometric plan (naive reference). The tumor is segmented on the undeformed CT, a uniform safety margin of fixed thickness is added, and the cutting path is executed open-loop with scripted retraction. This mirrors the CT-derived planner in the supervised autonomous prostate enucleation framework, which reported 97.1% resection accuracy on hydrogel phantoms in a different organ. The margin thickness is swept to trace a curve of remaining volume against positive-margin rate. Our method beats it if, under accurate-model scoring, its curve lies strictly above at every $\delta$ in the range of interest.

Crude-model-only reinforcement learning. This uses the same constrained policy optimization, demonstration seeding, compute, and tuning budget, with no accurate feedback. It isolates whether the crude model's margin errors matter at all. If it matches our method under accurate-model scoring, the whole premise fails.

Synchronous multi-fidelity training. This is the strongest competitor and a real threat. Every $k$ learner updates, training blocks while a batch of trajectories is scored by the finite-element model, and the residual model is refit. It uses the same evaluators, the same number of accurate evaluations, and the same wall-clock budget. Our method beats it if it reaches the target constraint at equal remaining volume in less wall-clock time, or achieves better volume at equal time.

Ablations. Two ablations isolate the key ingredients. The first is the split system without the ratio correction, which uses stale labels raw. The second replaces uncertainty-based selection with random selection of trajectories for evaluation. A further variant replaces the variable-latency accurate evaluator with a fixed-latency one, to see whether asynchrony helps only because of stragglers.

Reported numbers. No reported result was found on Lacuna for any learned policy on kidney tumor excision. The nearest reported learned-manipulation figures are on a different, easier deformable task: TissueRetract Task III success of 80% for pure DDPG and 76% for Cycle-of-Learning with 100 demonstrations (TissueRetract). Those figures calibrate expectations but are not comparable targets. All baselines on our task will be run by us.

Why not just…?

Why not just imitate surgeons? No kidney resection demonstration dataset appears in the material. The 211-demonstration prostate set in the supervised autonomous prostate enucleation framework covers retraction only, with cutting left to a static plan. In TissueRetract, pure imitation (SQIL) failed when tissue properties varied, whereas reinforcement learning recovered from elastic recoil. Demonstrations remain useful as a seed, and we use scripted ones that way. They do not replace learning against a margin cost.

Why not just run everything on one GPU, as Surgical Gym does? We do this for the actors. The crude model is GPU-vectorized, and the separation we propose is between the crude simulator and the finite-element evaluator, not across many machines. Surgical Gym's 344,828 FPS figure applies only to rigid bodies (Surgical Gym). Accurate cutting with element removal has data-dependent cost that would stall a lockstep batch. The fixed-latency ablation tests whether this straggler argument actually matters.

Why not just refit the correction synchronously every few updates? This may work as well. If the synchronous multi-fidelity baseline matches our method at equal wall-clock time, the asynchrony contributes nothing, and the contribution reduces to learning the residual cost. The delay sweep settles the question. If the policy changes little between evaluations, blocking costs little. The asynchronous design should win only when evaluator latency is long and uneven relative to learner update speed. We will report that regime boundary rather than assume it.

Why not use model predictive control on the finite-element model? A nonlinear solve with changing cut topology at every control step is far slower than the update rates of tens of milliseconds seen in the kidney surgery workflow system (real-time RAPN phase recognition, 16.65 ms end-to-end). Model predictive control on the crude model with the learned residual cost is a fair hybrid. We include it as a secondary comparison, because if it wins, the residual model is the contribution and the policy is not.

Proposed Experiments

We train on the 210 KiTS19 training cases and test on held-out cases, stratified by tumor size and by how deeply the tumor sits inside the kidney. Every method is scored only by the accurate finite-element model. The scores are positive-margin rate, remaining healthy volume as a fraction of the preoperative kidney, ischemia steps used, and wall-clock and evaluator-hours to reach $C \le \delta$ for $\delta \in \{1\%, 5\%\}$.

Three results decide the proposal.

First, crude-model-only training must show a measurable positive-margin rate under accurate scoring that the residual correction removes. Otherwise the premise is false.

Second, the split system must beat synchronous multi-fidelity training at equal evaluator budget in the long, uneven latency regime. Raw stale labels must measurably degrade constraint satisfaction as delay grows, and the ratio correction must recover it. A null result on either point would be reported as such.

Third, under perturbed tumor masks, the learned policy should widen margins where the tumor sits deep and keep them tight elsewhere. A uniform margin cannot do this, and this behavior is what would make the remaining-volume gain clinically meaningful rather than a simulator artifact.