Endoscopic submucosal dissection (ESD) is a procedure in which an endoscopist removes an early tumor by cutting through the thin, loose layer beneath the gut lining, staying above the muscle wall. Cutting too deep perforates the organ. Deep-learning systems now try to help with this. Some recognize which step of the operation is underway, some outline the "safe zone" to cut, and some predict a continuous safety margin or a knife trajectory. These systems are judged by a single accuracy or overlap score, computed once on one machine. In practice, though, a model trained on a data-center GPU will be run on a hospital PC or an embedded robot controller, at lower numerical precision, under different library versions, and with prompts drawn by different surgeons. Work on computational reproducibility shows that such "incidental" changes can move results by as much as the gains papers claim over each other. Nobody has measured this for ESD guidance.
We propose a decision-level reproducibility audit for ESD models. We rerun open ESD models under a crossed design that varies the random seed, the hardware and precision, the library stack, and the human prompter. We then measure, frame by frame and pixel by pixel, how often the model's clinically relevant decision changes. Examples are a pixel switching between "safe to cut" and "not safe", a safety-margin value crossing a threshold, or a phase label flipping. Finally, we attribute those changes to each factor. The two-sentence version is this. Aggregate scores like mean overlap may barely move across machines, while the specific boundary pixels and phase transitions a surgeon or robot acts on may move a lot. We measure how much they move, which part of the execution setup causes it, and whether that variation is larger than the published differences between competing ESD methods.
Core Claim
Our hypothesis has two parts. First, for current ESD guidance models, the disagreement between runs that differ only in seed, hardware, precision or software version is concentrated at decision-relevant locations: dissection-zone boundaries, safety-margin thresholds, and phase transitions. Aggregate metrics hide this concentration. Second, the combined seed and platform variation in headline metrics is of the same order as reported method-to-method margins. The sharpest case is SPRMamba, which reports 87.64% accuracy on ESD385 against 86.64% for SAHC, a one-point gap measured without any reported run-to-run variance.
What is new is not the use of a variability statistic. It is three things together. We move the unit of analysis from the benchmark score to the individual surgical decision. We decompose disagreement across the full set of experimental variables rather than seeds alone. And we separate computational variation from the optical-device variation that ESD papers already study. If the hypothesis is wrong, and platform and precision changes flip almost no decision-relevant pixels or frames, then that negative result is itself clinically useful. It would license porting these models to edge hardware without per-device revalidation.
Prior Work and Where This Sits
Three ESD lines motivate the audit. All three report point estimates on a fixed execution setup.
PDZSeg adapts a DINOv2 vision transformer with low-rank adapters (LoRA, a small set of added trainable weights) to segment the dissection zone on ex vivo porcine robotic ESD. It encodes surgeon prompts as colored overlays and stress-tests the model against simulated smoke, blur and motion. It does not test prompt-drawer variability or execution platforms.
ETSM predicts a continuous safety-margin map with RCMNet and a cutting trajectory with behavior cloning. It releases code and data, but it never asks how the continuous map drifts when ported.
AI-Endo recognizes surgical phases in real time at 47 fps. It shows transfer from Olympus to Fujifilm endoscopes, and it introduces the Normalized Transition Index, a phase-switching rate used to score operator hesitation. It runs all inference on one workstation. That last point matters for us. If the model's own flicker changes with the platform, then a skill score built on transition counts partly measures the computer rather than the surgeon.
Ebigbo et al. segment vessels and tissue layers during third-space endoscopy, smooth outputs with an exponential moving average, and see vessel detection drop from 95% in esophageal to 70% in rectal ESD. Their work is a reminder that domain shifts already cost a great deal. Our audit must therefore measure computational variation against that optical yardstick rather than in isolation.
The reproducibility literature supplies both the method and the warning. The method comes from two papers. Freire and Chirigati break an experiment into platform, research objective, implementation, method, actors and data (the PRIMAD model). We adopt this directly as our factor design, with surgeons as "actors". Belz et al. define Quantified Reproducibility Assessment around a small-sample-corrected coefficient of variation, $CV^*$. They show that even changing only the evaluation script on identical outputs gives $CV^*$ up to 4.176. We use $CV^*$ for aggregate metrics.
The warning comes from four papers. Popovic et al. find that seed changes alone swing BLEU from 84.47 to 89.59, and that hardware and library versions matter comparably. Tustison et al. show identical neuroimaging pipelines diverging across operating systems through hidden defaults. Pocock et al. document floating-point changes from minor runtime updates. Ernst et al. observe solver and version changes flipping a safety verdict.
That last finding is the closest analogue to our concern, and it is also why we track decisions rather than averages. One limitation of the metrology work constrains our design. $CV^*$ needs a ratio scale with a meaningful zero. It is therefore poorly suited to per-pixel boundary shifts, so we pair it with direct disagreement rates.
Research Program
Models and data. We work with models whose code and data are available. These are PDZSeg on the released ESD-DZSeg benchmark (1,849 images from 19 ex vivo porcine procedures), and RCMNet plus the behavior-cloning trajectory model on the ETSM dataset. For phase recognition, we reimplement an SPRMamba-style state-space model and a TCN-plus-transformer model in the AI-Endo style on Cholec80, and on ESD385 if the Renji Hospital data can be obtained. We include phase recognition specifically because state-space scans and long temporal convolutions accumulate numerical error over hour-long sequences in ways that single-frame segmenters do not.
Crossed factor design. Each model is trained under several seeds. Each trained model is then executed on a grid of conditions. On the platform side, the grid covers a data-center GPU, a consumer GPU, an embedded GPU of the kind used in robot controllers, CPU-only execution, and a Raspberry Pi with an Intel Neural Compute Stick of the type benchmarked in the Sensors 2025 study. Each is run at 32-bit, 16-bit and, where supported, 8-bit precision. On the implementation side, the grid covers two deep-learning library and driver versions. For PDZSeg, it also covers the actor factor: five people with endoscopy annotation experience independently draw scribble and box prompts on a fixed test subset. Containers are held fixed within each cell, so that what we measure is variation that survives containerization.
Decision-level measures. For the zone segmenter, we report three quantities. The first is the pairwise pixel flip rate inside a narrow band around the ground-truth zone boundary, compared with the flip rate in the interior. The second is the 95th-percentile boundary displacement in pixels. The third is the fraction of frames where any predicted-safe pixel moves onto annotated non-safe tissue.
For the safety-margin regressor, we report the rate at which a pixel's predicted margin crosses a fixed operating threshold between conditions. For trajectories, we report endpoint displacement between conditions, set beside the reported 9.10-pixel average displacement error.
For phase recognition, we report the frame-level label flip rate, the change in transition count per procedure, and the change in a transition-rate index computed as AI-Endo defines it.
Attribution. We fit a random-effects model to each decision-level outcome, with crossed effects for seed, platform-by-precision, library version and prompter. This apportions output disagreement across factors. We then compare each component against two reference magnitudes. The first is the reported method-to-method gap. The second is the change caused by the visual degradations PDZSeg already simulates, such as smoke and blur. The comparison gives an "optical versus computational" ratio, which tells clinicians which source of variation actually deserves their validation effort.
Mitigation. If platform variation turns out to matter, we test two cheap fixes. One is platform-paired validation, meaning thresholds re-tuned on the target device. The other is precision-aware fine-tuning. We report how much each one shrinks the platform component.
Baselines
Status quo. The naive reference is current practice: a single trained model, run once on the training workstation, scored by its headline metric. The published numbers are the anchors here. PDZSeg reports mean IoU 86.60% and Dice 91.93% with long-scribble prompts. RCMNet reports MAE 3.18. Behavior cloning reports ADE 9.10 pixels. SPRMamba reports 87.64% on ESD385 and 93.60% on Cholec80. We first reproduce these on the original setup. Our audit counts as informative relative to the status quo only if at least one execution condition that a hospital would plausibly use produces decision-level disagreement that the headline number does not reveal. One example would be a boundary flip rate well above the interior flip rate, while mean IoU stays within a point.
Seeds only. The strongest existing reporting practice, mean ± standard deviation over seeds, is run with the same trained checkpoints and an identical seed count. This is the real competitor, because it is cheap and already accepted. The full audit beats it if the non-seed components (platform, precision, version, prompter) together explain a substantial share of decision-level disagreement that seed variance does not predict. A further test: ranking deployment conditions by seed variance alone should fail to identify the conditions with the most boundary flips.
Containerized deterministic execution. A fixed container with deterministic library flags gives bit-exact repeatability on one machine. We run it on every platform. The audit beats this baseline if disagreement persists across devices despite identical containers.
Aggregate-metric ablation. The ablation that isolates our key ingredient replaces the decision-level measures with $CV^*$ computed on aggregate metrics over the same runs, with an identical factorial grid and an equal number of runs. If aggregate $CV^*$ flags the same problematic conditions as the decision-level measures, our main claim fails and the simpler QRA protocol of Belz et al. should be preferred.
Why not just…?
Why not just report mean and standard deviation over five seeds? Seeds cover only what Popovic et al. call arbitrary variation. Their own results show that incidental hardware and library changes are comparably large, and an ESD model deployed on a robot controller encounters exactly those changes. Seed variance on an aggregate also averages away where the disagreement sits. The seeds-only baseline above tests this directly. If seed variance already predicts boundary flips across platforms, we will say so and recommend seeds-only reporting.
Why not just ship the model in a Docker container? A container fixes libraries but not the processor's arithmetic. Freire and Chirigati note that it guarantees repeatability rather than portability across hardware, and Pocock et al. report output changes from runtime versions alone. The 8-bit and accelerator paths cannot even run the same kernels. The cross-device runs with identical containers settle this question empirically.
Why not just smooth the outputs over time? Exponential moving averages, used by Ebigbo et al., and dilated temporal convolutions, used by SPRMamba, suppress flicker. But they cannot tell algorithmic flicker from a real event, and smoothing delays alarms. Ebigbo et al.'s system still missed 2 of 9 bleeding vessels. Smoothing is a real threat to the phase-recognition part of the claim, so we run every phase experiment both with and without smoothing. If smoothing removes the platform component of transition-count changes without adding measurable lag at true transitions, it is the practical answer for phase models, and our contribution narrows to the per-frame segmentation and margin models, which have no temporal context.
Why not just use classic CNNs, which are mature and supposedly deterministic? They are not obviously more portable, since they face the same floating-point and precision issues, and they cost accuracy: ResNet-50 reaches 72.31% on ESD385 against 87.64% for SPRMamba. We include ResNet-50 in the grid to test whether architecture family changes the platform component at all.
Decisive Checks
Three checks decide the project.
The first asks whether, on ESD-DZSeg, boundary-band pixel flips across platform-by-precision cells are clearly larger than interior flips, and are detectable while mean IoU varies by less than a point. If they are not, the concentration hypothesis is false.
The second asks whether the combined seed and platform spread in phase accuracy on Cholec80, and on ESD385 if available, is comparable to the one-point SPRMamba–SAHC margin. If it is, published rankings on this benchmark cannot be trusted without variance reporting.
The third asks whether the transition-count index changes across execution conditions by an amount comparable to the junior-versus-expert difference it is meant to detect. AI-Endo reports no such per-group figure that we could find on Lacuna, so we will compute it from our own runs. If it does, skill reports built on such indices need platform locking.
Each check has a clear negative outcome, and each negative outcome is a usable deployment guideline in its own right.
References
- SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba
- PDZSeg: adapting the foundation model for dissection zone segmentation with visual prompts in robot-assisted endoscopic submucosal dissection
- ETSM: Automating Dissection Trajectory Suggestion and Confidence Map-Based Safety Margin Prediction for Robot-Assisted Endoscopic Submucosal Dissection
- Intelligent surgical workflow recognition for endoscopic submucosal dissection with real-time animal study
- Vessel and tissue recognition during third-space endoscopy using a deep learning algorithm
- Provenance and the Different Flavors of Reproducibility
- Quantified Reproducibility Assessment of NLP Results
- Exploring Variation of Results from Different Experimental Conditions
- Instrumentation bias in the use and evaluation of scientific software: recommendations for reproducible practices in the computational sciences
- Provenance Design and Evolution in a Production ML Library
- ARCH-COMP 2021 Category Report: Falsification with Validation of Results
- Acceleration of Image Classification and Object Tracking by the Intel Neural Compute Stick 2 with Power Efficiency Evaluation on Raspberry Pi 4B