Off-Policy Evaluation of Display Alignment in Decoupled Surgical Viewing

Surgeons who look through a microscope's eyepieces must bend their neck and spine to follow the instrument's optical axis. Digital exoscopes and heads-up 3D displays remove that constraint: the camera can point anywhere while the surgeon sits upright and watches a screen or headset. The cost is a new mismatch. The hands now move in one coordinate frame and the eyes watch another, which may be rotated, tilted or mirrored relative to the hands. Any display system can partly undo this by rotating or re-orienting the image, or by re-aiming the camera. Each such correction trades hand-eye agreement against other costs, such as losing the anatomical "up" direction, slewing the camera mid-task, or pushing the operator back toward a strained posture. Today nobody can say which correction rule is best without running a new user study for every candidate.

We propose to answer that question from logged sessions instead of new trials. Operators run surgical training tasks while the system applies randomized, recorded display corrections. A value estimator fit to those logs then predicts how well any other correction rule would have done, scoring task success, hand-eye misalignment and a posture penalty. This is off-policy evaluation, which means estimating the value of one decision rule from data generated by a different one. Surgical sessions are long, so the estimator must bootstrap, updating each value guess from the guess at the next time step. Combined with function approximation and off-policy data, this is exactly the "deadly triad" under which ordinary temporal-difference learning can diverge. We therefore use gradient-corrected two-timescale temporal-difference learning with linear features. Its stability under realistic, correlated logged data has recently been proven. We test whether that guarantee actually matters on surgical teleoperation logs, where it has never been checked.

Core Claim

The first claim is about the problem. Choosing between optical-postural coupling and decoupling is not a binary hardware decision. It is a sequential control problem over display transforms, and its candidate policies can be ranked from randomized logs before any human trial. The second claim is about the method. On such logs, which are long-horizon, collected under a different policy, and drawn from single correlated trajectories, plain off-policy semi-gradient TD will sometimes blow up. TDC(λ), the gradient-corrected method analyzed by Convergence of Two-Timescale Markovian Stochastic Approximations with Applications in Reinforcement Learning, stays bounded and ranks policies as well as or better than non-bootstrapping and batch alternatives. Both claims can fail. If a fixed "always align the image to the hands" rule wins regardless of context, the decision problem is trivial. If off-policy least-squares TD or Monte Carlo importance sampling matches TDC, the stability machinery is unnecessary. The experiments are built to detect both outcomes.

Where This Sits

The clinical literature motivates the problem but does not model it. Microscope eyepieces force sustained hunched postures during vitrectomy (3D Heads-Up Display vs. Standard Operating Microscope Vitrectomy for Rhegmatogenous Retinal Detachment). Exoscopes let surgeons work upright in retrosigmoid, deep basal-ganglia and gravity-retraction corridors (Retrosigmoid Approach in the Supine Position Using ORBEYE, Surgical resection of glioblastoma in basal ganglia and utility of exoscope, Utility of a novel exoscope, ORBEYE, in gravity-assisted brain retraction surgery). Even robotic consoles retain head-down coupling, which led to a virtual-reality viewer that frees head motion (A multimodal virtual vision platform as a next-generation vision system for a surgical robot). These works describe hand-eye dissociation as a learning curve, and none of them proposes a rule for correcting it. Skills also transfer poorly between viewing modalities (Robotic-assisted versus laparoscopic bowel anastomoses). That finding suggests the mismatch is not reliably absorbed by practice, and it is part of the motivation.

The surgical-learning platforms supply the testbed but not the method. SurRoL (SurRoL) includes endoscope-camera tasks such as StaticTrack. Surgical Gym (Surgical Gym) adds Active Tracking and Endoscopic Image Matching at high simulation throughput. dARt Vinci (dARt Vinci) records headset-based teleoperation at 72 Hz, including the operator's head pose, which we use as a posture signal. All three assume a fixed or canonical camera-to-tool relation and train with deep RL. None evaluates alternative viewing policies. ASHA (ASHA) adapts the mapping from noisy human input to robot action. We do not touch the motor mapping. We evaluate what the operator is shown. SSP (SSP) adds a barrier-function safety filter to learned policies. That design is complementary, since our target policies act only on the display and cannot cause collisions.

The stability literature supplies the method and sets its limits. Tsitsiklis and Van Roy (Analysis of Temporal-Diffference Learning with Function Approximation) prove that linear TD converges on-policy and note that this guarantee fails off-policy. Maei and colleagues (Toward Off-Policy Learning Control with Function Approximation, Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation) introduce the gradient correction. That correction uses a second weight vector, updated faster, that tracks the projected Bellman error. Mahadevan et al. prove that TDC(λ) stays bounded and converges almost surely under Markovian noise without projection boxes. This is the property we need, because each surgical log is one correlated trajectory rather than independent samples. Wang et al. (Non-Asymptotic Analysis for Two Time-scale TDC) give finite-sample rates that guide our step-size choices. Xie and Jiang (Batch Value-function Approximation with Only Realizability) constrain the design: a linear model can be badly misspecified. We therefore treat feature richness as an experimental variable rather than an assumption. As far as the file shows, none of these stability results has been tested on physical or kinematic teleoperation data.

Research Program

Environment. We build a viewing layer on top of SurRoL's camera-manipulator tasks and NeedlePick/PegTransfer, mirrored in Surgical Gym for fast simulation. At each decision point, every 0.5 s, the layer chooses a display transform. The choice set covers image rotation in 30° steps, a mirror flag, and an optional camera re-aim toward the axis of the tool shaft.

Logging (behavior) policy. During logging, transforms are drawn from a known randomized behavior policy $\mu$ with full support. Exact importance ratios $\rho_t = \pi(a_t \mid s_t)/\mu(a_t \mid s_t)$ are therefore available, where $\pi$ is a candidate target policy, $a_t$ the transform chosen at step $t$ and $s_t$ the state. This avoids estimating a human behavior model.

Reward. The per-step reward combines three terms: task progress from the simulator, minus a cost on the angle between the hand-motion frame and the displayed frame, minus a posture cost. The posture cost is computed from headset head pitch in dARt Vinci-style collection or from monitor gaze angle in a heads-up-monitor condition. Running both display media also lets us measure the headset-versus-monitor tension raised by Upgrading Instruments for Robotic Surgery.

Features and target policies. The linear features are sines and cosines of the misalignment angle, tile-coded tool-to-target distance, task phase, head pitch, and their pairwise products. The target policies are always-align, never-align, half-align, posture-capped align, and a phase-dependent rule that aligns during fine manipulation and keeps the anatomical up direction during navigation.

Data sources. Logs come from two sources. The first is a simulated operator whose motor error grows with misalignment angle and whose posture follows the display, with parameters varied across a family of operators. Here every target policy can also be rolled out on-policy to obtain true values. The second is a pilot cohort of human operators on the VR teleoperation setup, where we hold out a few target policies and run them live for validation.

Baselines

The naive reference point is to report the average logged return under the randomized behavior policy and to deploy whichever single fixed transform appeared most often in successful episodes. This represents doing nothing beyond descriptive statistics. Any evaluator worth having must rank policies better than this frequency heuristic.

The strongest bootstrapping-free competitor is per-decision importance-sampled Monte Carlo evaluation. It uses the same exact $\rho_t$, the same logs, and no value function. It is unbiased and immune to the deadly triad, but its variance grows with horizon. We also run GradientDICE (GradientDICE), which estimates stationary-distribution ratios instead of per-step products, using the same features.

The strongest batch bootstrapping competitor is off-policy LSTD(λ), which solves the same fixed point as TDC in closed form. Because it matches TDC's solution when well-conditioned, it is the baseline most likely to tie us. We also run deep fitted Q-evaluation with a small MLP to check whether nonlinear capacity, rather than stability, is what matters.

The key ablation is off-policy semi-gradient TD(λ). This is the same features, step sizes and λ as TDC, with the gradient-correction term deleted. It isolates exactly the ingredient the proposal depends on. A second ablation removes the misalignment-by-phase interaction features to test the realizability concern.

All methods receive identical logs, identical features (except deep FQE), and the same tuning budget of 50 configurations. Tuning is selected on simulated operators that are disjoint from the test operators. For context, the closest task-level numbers in the file are 90% success on SurRoL StaticTrack and 96% on GauzeRetrieve under fixed cameras (SurRoL). No reported result was found on Lacuna for off-policy evaluation of viewing or camera policies on SurRoL, Surgical Gym or dARt Vinci logs.

Beating the baselines means two things. The first is a higher Spearman rank correlation between estimated and true policy values across held-out simulated operators than LSTD and importance sampling, with non-overlapping bootstrap intervals. The second is zero divergent runs, defined as weight norm exceeding 10³ times its initial value, where semi-gradient TD shows a nonzero divergence rate.

Why not just…?

Why not just apply the known rigid-body rotation and always align the image to the hands? When camera and tool poses are known, perfect alignment is one matrix multiplication. That is why always-align is one of our target policies rather than something we argue against. The open question is whether alignment is always worth its side effects: losing gravity or anatomical orientation, camera re-aiming that interrupts the task, and posture that follows a rotated display. If always-align comes out on top across operators and both display media in the simulated ground truth and the human validation, we will report that the trade-off is degenerate, and that result is useful in its own right. The evaluator is only valuable if some context-dependent policy beats the fixed rule.

Why not just use importance-sampled Monte Carlo and avoid bootstrapping entirely? This is a real threat for short tasks, and we expect it to tie on NeedleReach-length episodes. Per-step ratio products grow multiplicatively with horizon, so on multi-minute PegTransfer sessions the variance should dominate. We vary episode length explicitly and report the horizon at which the two methods cross over. If no crossover appears within realistic lengths, the stability argument loses its practical force.

Why not just solve off-policy LSTD in batch? LSTD shares TDC's fixed point, so on well-conditioned data it should match TDC, and we would say so. TDC's claimed advantage is narrower: it works as streaming updates on continuously arriving logs, it has no matrix inversion as features grow, and its proven boundedness holds when the off-policy LSTD matrix is near-singular. We measure the condition number of the LSTD matrix across policies and report error as a function of it. If LSTD is never ill-conditioned on these logs, TDC's case rests only on the streaming setting.

Why not just run a user study for each candidate policy? For five policies, a crossover study is feasible, and our human validation is exactly that, done on a subset. The point of off-policy evaluation is to screen many policies and parameter settings, for example posture caps and phase thresholds, from one randomized dataset. The human study checks whether the off-policy ranking predicts live outcomes. If it does not, the program fails at its main purpose.

Decisive Experiments

Experiment 1: does the triad bite? On simulated-operator logs, we run off-policy semi-gradient TD(λ) and TDC(λ) for target policies at increasing distance from $\mu$, measured as the expected squared importance ratio. We report divergence rate and weight-norm trajectories. The proposal requires semi-gradient divergence to be non-negligible at realistic policy distances. If it never diverges, the method half of the claim collapses, and we fall back to the cheapest estimator that ranks correctly.

Experiment 2: ranking accuracy. Across held-out simulated operators, we compare the Spearman correlation and normalized absolute value error of every baseline against on-policy ground truth. This is repeated for heads-up-monitor and headset posture models and under the reduced-feature ablation, which shows how much misspecification costs.

Experiment 3: transfer to people. We collect randomized logs from the human cohort, rank the target policies off-policy, and then run the top-ranked, bottom-ranked and always-align policies live. Outcomes are task success, completion time and head-pitch posture cost. The decisive result is agreement between the off-policy ranking and the live ranking. A context-dependent policy beating always-align live would confirm the problem claim. A tie would show that a fixed alignment rule is enough.

References

  1. Convergence of Two-Timescale Markovian Stochastic Approximations with Applications in Reinforcement Learning
  2. 3D Heads-Up Display vs. Standard Operating Microscope Vitrectomy for Rhegmatogenous Retinal Detachment
  3. Retrosigmoid Approach in the Supine Position Using ORBEYE: A Consecutive Series of 14 Cases
  4. Surgical resection of glioblastoma in basal ganglia and utility of exoscope: Technical case reports
  5. Utility of a novel exoscope, ORBEYE, in gravity-assisted brain retraction surgery for midline lesions of the brain
  6. A multimodal virtual vision platform as a next-generation vision system for a surgical robot
  7. Robotic-assisted versus laparoscopic bowel anastomoses: randomized crossover in vivo experimental study
  8. SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning
  9. Surgical Gym: A high-performance GPU-based platform for reinforcement learning with surgical robots
  10. dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scale
  11. ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning
  12. SSP: Safety-guaranteed Surgical Policy via Joint Optimization of Behavioral and Spatial Constraints
  13. Analysis of Temporal-Diffference Learning with Function Approximation
  14. Toward Off-Policy Learning Control with Function Approximation
  15. Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation
  16. Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximation
  17. Batch Value-function Approximation with Only Realizability
  18. Upgrading Instruments for Robotic Surgery
  19. GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values