Shrinking the Robustness Tax in Robotic Tissue Retraction by Watching the Tissue Deform

A robot trained in simulation to pull back soft tissue has to cope with the fact that nobody knows, before contact, how stiff a given patient's tissue is or how firmly it is anchored. The standard fix is domain randomization: train across many simulated stiffnesses so the policy works on all of them. This often produces timid behavior. The robot pulls gently, moves slowly, or gives up, because one fixed behavior has to be safe on the most fragile tissue and still succeed on the most resistant. Robust reinforcement learning calls this the robustness tax, the reward a policy gives up in typical conditions to stay safe in rare ones. Our argument is that in soft-tissue manipulation most of this tax comes from the policy not knowing which tissue it is touching, and much less from the choice of robust objective. Tissue gives that knowledge away cheaply: within the first second of contact, the way the visible surface moves under a known tool motion reveals its stiffness.

We propose to measure the robustness tax in tissue retraction, split it into an "objective" part and an "information" part, and remove the information part with a policy that keeps an explicit belief over tissue mechanics. The belief is updated from tracked surface displacement during the episode, and the safety constraint is enforced against the tail of that belief, not against the tail of the training distribution. As the belief narrows, the safety margin narrows with it, so caution is spent only while ignorance lasts. The test is concrete. On randomized-stiffness retraction tasks and on a da Vinci Research Kit (dVRK) with tissues of different stiffness, the belief-conditioned policy should close most of the gap to a policy that is told the true stiffness, at matched tissue-overload rates. It should also beat memoryless randomization, worst-case robust training, and simple probe-then-commit identification.

Core Claim

Let $\theta$ be the tissue parameters (stiffness, damping, anchoring strength). Let $J(\pi,\theta)$ be the task return of policy $\pi$ on tissue $\theta$, and let $J^\star(\theta)$ be the return of an oracle policy trained with $\theta$ known. We define the tax of $\pi$ at $\theta$ as $T(\pi,\theta)=J^\star(\theta)-J(\pi,\theta)$. We always measure it at a matched safety level: the rate at which peak tissue force exceeds a damage threshold $F_{\text{tear}}$. Standard domain randomization already maximizes average return, not worst-case return. Its conservatism therefore cannot be blamed on pessimism in the objective. It comes from a memoryless policy hedging across tissues it cannot tell apart.

Our claim is that a policy with an explicit posterior $b_t(\theta)$ over tissue parameters can remove most of $T$ without raising the overload rate. That posterior is computed from observed tissue displacement. The policy acts under the per-step constraint

$$\Pr_{\theta\sim b_t}\big[\,Q_c(s_t,a_t;\theta) > F_{\text{tear}}\,\big]\le \delta,$$

where $s_t$ is the observed state, $a_t$ the action, $Q_c$ a learned predictor of the peak tissue force that action will cause on tissue $\theta$, and $\delta$ the allowed risk. Early in contact, $b_t$ is broad and the constraint is as cautious as a robust policy. Once the tissue has responded to a few millimeters of tool motion, $b_t$ concentrates and the allowed actions approach the oracle's.

Where This Sits

Three lines of work motivate this. The first shows that the tax is real. Robust Planning with (L)RTDP shows that planning against interval uncertainty inflates expected cost. Fuz-RL shows that extreme pessimism can make an agent abandon its task entirely. Soft-Robust Actor-Critic Policy-Gradient replaces the worst case with an expectation under a fixed prior over dynamics and recovers performance on Cart-Pole and Pendulum. That prior is static. Ours is updated within the episode, which is the limitation that paper itself leaves open.

The second line shows that adaptation beats a single policy. Adaptive Policy Portfolios for Robust Markov Decision Processes cuts robust regret from 14.28 to 0.80 on datacenter control by selecting among $K=10$ policies with an upper-confidence-bound bandit. That bandit learns across episodes and treats each policy as a black box. Surgery gives one episode per patient, so identification has to happen within the episode, from the physics of the tissue. Model-Based Offline RL with Pessimism-Modulated Dynamics Belief puts pessimism into the dynamics belief rather than into a reward penalty. We take the same stance, and make the belief online and observation-driven.

The third line is surgical work, and it shows what is missing. Autonomous Soft Tissue Retraction Using Demonstration-Guided RL randomizes elasticity in the PyBullet environment TissueRetract. It identifies elastic recoil ("Elastic Pull Factor") as the central difficulty and reports 80% success on its hardest task with pure reinforcement learning. It never measures whether the randomization made the policy conservative, and it stays in simulation. Sim-to-Real Transfer for Visual RL of Deformable Object Manipulation uses image translation, a learned network that makes simulated frames look like real camera frames, to train a dVRK retraction policy in the SOFA simulator. It reaches 50% real success (13/32), with failures attributed to grasp and interaction mechanics, a dynamics gap that image translation cannot touch. Safe RL using Formal Verification for Tissue Retraction cuts verified safety violations from 27.02% to 3.07% at unchanged tumor exposure (~0.41) in UnityFlexML. Its verification is offline and assumes one fixed tissue model. Our proposal adds what these three omit: varying mechanics, an explicit tax measurement, and a safety constraint that adapts online.

Two works supply method. Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics couples dVRK stereo point clouds to a position-based dynamics simulator, a fast solver that moves mesh points to satisfy physical constraints. It tracks lifted chicken skin and pork while assuming fixed stiffness. We reuse it as the tracker, and we invert its assumption: the mismatch between tracked and predicted displacement becomes the evidence for updating stiffness. Asymmetric Actor Critic trains a critic on privileged simulator state while the deployed actor sees only real observations. We use the same idea to train $Q_c(s,a;\theta)$ with the true $\theta$, which is available in simulation. One work constrains the design. ZeMa transfers deformable grasping zero-shot at 70% without randomization, using a high-fidelity contact simulator. This tells us that simulator fidelity matters and that our gains must not just reflect a poor simulator. It also only works when the material is known in advance, which is exactly the condition surgery violates.

Research Program

We first build a parameterized retraction task in a simulator supporting both randomized mechanics and the dVRK geometry: SOFA, as in the 50% transfer work, with TissueRetract's task structure ported over. Stiffness, damping and anchor strength are sampled over a wide range spanning soft fat-like to dense tissue. Simulated per-node forces give ground-truth overload events.

Second, we measure the tax. We train oracle policies on a grid of $\theta$, then compute $T(\pi,\theta)$ for memoryless domain randomization at several randomization widths and for worst-case adversarial training, each tuned to the same overload rate. The first result is a curve of tax against randomization width, split by tissue stiffness. This curve does not exist for tissue manipulation, and it is useful whether or not our method works.

Third, we build the belief-conditioned agent. A particle filter, which tracks the belief as a set of weighted guesses, runs $N$ cheap simulated copies with sampled $\theta$. Each copy is driven by the commanded tool motion. Particles are reweighted by how well their predicted surface displacement matches the displacement tracked by the position-based dynamics registration. The actor receives the observation plus summary statistics of $b_t$ (mean and quantiles). Actions violating the posterior chance constraint above are projected to the nearest feasible action. Because tightening the belief relaxes the constraint, the policy is rewarded for moving in ways that reveal stiffness. We test whether it learns such probing motions, and compare them with a scripted probe.

Baselines

The naive reference point is memoryless domain randomization: a feedforward PPO policy trained across the full stiffness range, the setting of TissueRetract. The closest reported number is 80% success on TissueRetract Task III with DDPG (Autonomous Soft Tissue Retraction). We reproduce it at matched environment steps before anything else. The real-robot reference is the 50% (13/32) dVRK success from Sim-to-Real Transfer for Visual RL, which we re-run with the same training budget of $10^7$ steps.

The upper reference is the privileged oracle that sees true $\theta$. It defines zero tax, and it is the target our method must approach.

Worst-case adversarial training represents the robust-RL answer. The adversary picks the stiffness within the same range. Soft-Robust Actor-Critic represents expectation-based robustness under a fixed prior.

The strongest competitor is a recurrent domain-randomization policy: a memory-equipped policy trained across stiffnesses with the same chance-constrained (Lagrangian) cost and the same privileged cost critic. It can identify the tissue implicitly from its history. It shares our data, simulator, compute and hyperparameter search budget. The only difference is that our belief is explicit and physically computed, and the constraint uses its tail. There is no reported result on Lacuna for recurrent or belief-conditioned policies on tissue retraction.

A probe-then-commit baseline runs a fixed scripted lift, fits stiffness by least squares on displacement, and then executes the oracle policy for the fitted value.

Two ablations isolate the key ingredient. The first keeps the belief as an input to the policy but enforces the constraint against the prior tail, not the posterior. The second keeps the posterior constraint but feeds the actor no belief.

Beating the baselines means two things. In simulation, recovering at least half of memoryless randomization's tax at the same overload rate, while exceeding the recurrent policy on stiffness values held out from training. On hardware, exceeding the 13/32 success count with fewer force exceedances.

Why not just…?

Why not just train a recurrent policy under domain randomization and let it identify the tissue implicitly? This is the real threat, and the proposal is built so it can lose to it. Implicit identification may match our return in distribution. Our expectation is that it degrades on held-out stiffnesses and at different safety thresholds $\delta$, because its caution is baked into its weights rather than computed from a calibrated belief. The recurrent baseline and the first ablation settle this. If the recurrent policy matches us everywhere, the contribution reduces to the tax measurement and the evidence that the tax is informational. We would report that plainly.

Why not just poke the tissue once, estimate stiffness, and use the matching policy? Probe-then-commit is cheap and may work well on homogeneous tissue. It fails if one probe misestimates anchoring or if stiffness varies across the tissue. The portfolio results in Adaptive Policy Portfolios also suggest that committing to a single estimate is fragile. We test it directly. If it matches the full method, we would recommend it as the practical answer.

Why not just build a more accurate simulator, as ZeMa does? ZeMa reaches 70% real success only for a pre-calibrated elastomer. A perfect simulator still needs the patient's parameters, which are unknown before contact. The tracking drift under assumed stiffness reported in Real-to-Sim Registration shows this concretely. Our oracle-at-wrong-$\theta$ runs quantify the cost.

Why not just fix the visual gap with image translation? The 50% ceiling of Sim-to-Real Transfer for Visual RL, with failures traced to interaction mechanics, indicates the remaining gap is in dynamics, not appearance. We keep its image translation in all hardware arms, so any gain we show is in addition to it.

Proposed Experiments

In simulation we report three things. The tax curves across randomization width and stiffness. Success, mean time to target exposure, and overload rate over a held-out stiffness grid that includes values outside the training range. Belief calibration, meaning whether the true $\theta$ falls inside the posterior's 90% interval about 90% of the time. We also re-run the UnityFlexML protocol of Safe RL using Formal Verification with stiffness randomized. The question there is whether its 3.07% violation rate survives varying mechanics, and whether ours holds that rate at tumor exposure no lower than ~0.41.

On hardware, we run dVRK retraction on chicken skin and pork, following the registration setup of Real-to-Sim Registration, plus silicone phantoms at three stiffnesses fitted with a force sensor. We use 32 trials per arm, matching the reference study, and report success counts with confidence intervals and the rate of force exceedances.

The decisive check is a dose-response test. If the tax is informational, our method's advantage over memoryless randomization should grow with randomization width. Within each episode, the margin of caution should shrink in step with the posterior's width. If neither pattern appears, the core claim is wrong.