Training-Only Smoke Physics for Surgical Instrument Segmentation

During laparoscopic surgery, electrocautery tools burn tissue and fill the abdomen with smoke. The smoke hides instrument tips and organ edges, which are exactly the regions a segmentation model must outline precisely. Current practice handles this in two steps: a separate network removes the smoke, and a segmenter then runs on the cleaned frame. That design adds latency to a pipeline that must run live. It also passes along any artifacts the cleaner introduces. A recent benchmark found that restorers scoring high on pixel fidelity can still hurt downstream instrument segmentation and depth estimation (Benchmarking Laparoscopic Surgical Image Restoration and Beyond).

We propose a single segmentation network that learns about smoke only during training. Small auxiliary heads predict how much light the smoke blocks at each pixel. They split this into a diffuse cavity-wide haze, which grows with distance from the camera, and localized plumes, which do not. The heads are deleted before deployment, so the deployed model is an ordinary segmenter with no added inference cost. The bet is that forcing the shared features to explain the smoke physically keeps instrument boundaries intact behind the smoke. We expect this to beat both desmoke-then-segment and plain smoke augmentation, and to show up most clearly in boundary-distance metrics.

Core Claim

Outdoor fog detection has shown that physics can serve as a training-time regularizer. Object Detection in Foggy Scenes by Embedding Depth and Reconstruction into Domain Adaptation adds three training-only heads to a Faster R-CNN detector: a depth head, a loss tying predicted fog transmission to depth, and a decoder that reconstructs the clean background. It reports 47.6 mAP on Foggy Cityscapes against a prior 44.3, with zero inference overhead.

That paper supplies our method, but it cannot be copied into surgery. It assumes one scattering coefficient for the whole image. The surgical smoke literature contradicts that assumption. Attention-Aware Laparoscopic Image Desmoking Network with Lightness Embedding and Hybrid Guided Embedding argues that surgical smoke density does not track camera depth and abandons the scattering model altogether. Rethinking Surgical Smoke instead shows that smoke separates into two kinds: turbulent "diffusion" plumes at the tool–tissue contact, and dispersed "ambient" haze across the cavity.

Our claim reconciles the two papers. The ambient part does obey a depth-dependent scattering law. The plume part is a sparse, localized correction. A segmenter trained with this decomposed physical constraint, rather than with a global-coefficient constraint or no physics at all, should segment instruments more accurately in real smoky video. The new element is the decomposed transmission constraint used as a training-only regularizer for segmentation. Neither the fog-detection work nor any surgical desmoking work does this. All surgical methods we found either restore images as a separate stage or ignore depth entirely.

The Physical Model

Fog and haze follow the standard scattering equation $I(x) = J(x)t(x) + A(x)(1-t(x))$. Here $I$ is the observed frame, $J$ the smoke-free scene, $A$ the light scattered back toward the camera, and $t(x)$ the transmission, meaning the fraction of scene light that survives to the sensor at pixel $x$.

Outdoors, $t(x)=e^{-\beta d(x)}$, where $\beta$ is a single scattering coefficient and $d$ is depth. We replace this with

$$t(x) = \exp\big(-\beta_a\, d(x) - \tau_p(x)\big),$$

where $\beta_a$ is a per-frame scalar for ambient haze and $\tau_p(x)\ge 0$ is the optical thickness of local plumes, meaning how much light they absorb along each ray. Because the laparoscope's light source sits beside the lens, we let $A(x)$ be a smooth, low-resolution color field rather than a single constant. This is a modeling choice we will test, not something established in the literature we draw on.

The auxiliary heads predict $t$, $\beta_a$, $\tau_p$, $A$ and a clean frame $\hat J$. Training combines four losses. A consistency term asks $-\log t$ to match $\beta_a d + \tau_p$. A sparsity penalty on $\tau_p$ prevents the plume term from absorbing everything. A re-rendering loss asks $\hat J t + A(1-t)$ to reproduce $I$. The segmentation loss itself is applied only to the mask output.

Research Program

The backbone is a standard instrument segmenter. The physics heads branch from its intermediate features and are removed at test time. We will run the same heads on two backbones, a Mask2Former-style region model in the spirit of MATIS and a frame-level U-Net-style model, to check that the effect does not depend on one architecture.

Depth supervision comes from an off-the-shelf monocular depth network applied to clean frames. No endoscopic depth model is benchmarked in our sources, so we will report which one we use and test sensitivity to it.

Supervision for the physics heads comes from three sources.

The first is synthetic smoke rendered with our two-part model onto clean, mask-labeled frames from EndoVis 2017/2018 and Robust-MIS 2019. Here $t$, $\beta_a$ and $\tau_p$ are known exactly. The second is the paired synthetic and real sequences of STSVD from Rethinking Surgical Smoke. STSVD's pixel annotations of plume versus ambient smoke directly supervise where $\tau_p$ should be nonzero. The third is real smoky frames from LSVD (Self-Supervised Video Desmoking for Laparoscopic Surgery) and SurgClean. Both pair a smoky frame with a motion-aligned clean frame captured just before smoke appeared, which supplies a real target for $\hat J$ without any rendering.

The synthetic source carries a real risk of circularity. A model trained on our renderer can learn our renderer rather than real smoke. Every headline result will therefore be measured on real smoke only.

For those real tests, we will build an instrument-mask set on several hundred smoky frames from LSVD and SurgClean. Masks are first propagated from the aligned clean frame and then corrected by hand. None of the existing desmoking benchmarks provide segmentation labels on real smoke at this scale. We will release this test set, which is useful independent of whether the method works.

Baselines

The naive reference point is the backbone trained on clean labeled data and evaluated directly on smoky frames. This measures how much smoke actually hurts. If the drop is small, the problem does not merit the method.

The strongest existing clean-frame segmenters set a ceiling for the backbones. MATIS reports 84.26% mIoU on EndoVis 2018. SurgicalSAM adapts a frozen Segment Anything encoder with a small 4.65M-parameter prompt module and reports 80.33 Challenge IoU there. Our model must stay within about one point of its own backbone on clean EndoVis frames, or the physics heads are trading clean accuracy for robustness.

The strongest competitor on the actual question is the two-stage pipeline. It runs STANet, the smoke-type-aware desmoker from Rethinking Surgical Smoke, followed by the same segmenter. As a lighter alternative it runs SelfSVD-S, which reports 0.03 s per frame with 1.92M parameters. STANet reports 33.53 dB PSNR on STSVD and a downstream instrument segmentation gain stated as "+0.587 IoU" over competing restorers. The scale of that figure is ambiguous in our source, so we will rerun it rather than quote it as a target.

For fairness, the downstream segmenter in the two-stage pipeline will be fine-tuned on desmoked outputs of the same training frames our model sees, with the same tuning budget. We will report end-to-end latency for both systems. Beating this baseline means higher boundary accuracy, measured as 95th-percentile Hausdorff distance (HD95) and average surface distance, on the real smoky test set. Matching Dice/IoU while beating boundary distance at lower latency also counts.

We found no reported instrument HD95 or ASD under real surgical smoke for any method in our sources. That gap is part of what we fill.

Three ablations isolate the key ingredient, and all use identical synthetic data and identical augmentation. The first adds a generic auxiliary head that reconstructs the clean frame with no physical structure. The second ports the fog-detection method directly, with a single global $\beta$. The third uses the decomposed model but drops the depth coupling, so $t$ is free and only re-rendering constrains it. The full model must beat all three for the claim to stand.

On a semi-supervised side setting, SegMatch's adversarial-augmentation result on Robust-MIS 2019 is 73.3 mean Dice with 10% labels, against 62.1 supervised (SegMatch). We will test whether our heads add to SegMatch rather than compete with it.

Why not just…?

Why not just augment training with rendered smoke and skip the physics heads? This is the most serious threat, because our method needs the same rendered smoke anyway. If invariance to rendered smoke is all that matters, augmentation alone should match us. The evidence against it is indirect. SegMatch found trainable perturbations beat handcrafted ones, but its perturbations were not physical, so that finding does not settle this case. The comparison is therefore built into the design: an augmentation-only model trained on exactly our rendered frames. If it matches the full model on real smoke, we will report that physically rendered augmentation is sufficient, which is itself a useful negative result.

Why not just desmoke first and then segment? Desmoking first is the current practice, and STANet makes it strong. Our concern rests on the SurgClean finding that higher-fidelity restoration does not reliably improve downstream parsing (Benchmarking Laparoscopic Surgical Image Restoration and Beyond). It also rests on latency: a 30 ms desmoker alone consumes a real-time frame budget. The fine-tuned two-stage baseline decides this directly. If two-stage wins on boundaries, the latency argument alone does not justify our method.

Why not just reuse the outdoor fog method with one global coefficient? That is our second ablation. The L-SAHGNet observation that smoke does not track depth predicts it will underperform, especially near active instruments where plumes form. To test that prediction, we will report results split by distance from STSVD's annotated plume regions.

Why not just fine-tune a foundation model on smoky frames? SurgicalSAM itself reports that SAM is sensitive to prompt jitter and visual ambiguity in the surgical cavity, so robustness is not free. We will fine-tune SurgicalSAM on our augmented data as an additional baseline and attach our heads to its tunable components where the architecture permits.

Proposed Experiments and Decisive Checks

The primary test set is the real smoky instrument-mask set built from LSVD and SurgClean. Metrics are IoU, Dice, HD95, ASD and frames per second. Secondary tests are STSVD with its downstream instrument protocol, and clean EndoVis 2017/2018 and Robust-MIS 2019 to measure any clean-frame cost.

Three checks are decisive.

First, the full model beats both the augmentation-only and the generic-reconstruction ablations on real smoke, with the gap widening as smoke gets denser. We will bin density using the dark-frame-referenced transmission estimated from the paired clean frames.

Second, the decomposed model beats the global-coefficient port specifically near plumes. Ambient-only regions should show little difference between them.

Third, the predicted ambient coefficient $\beta_a$ should correlate with STSVD's annotated ambient-smoke extent on held-out real sequences. This shows the heads learned something physical rather than an arbitrary auxiliary target.

If the first check fails, the proposal reduces to a dataset contribution and a careful negative result about physics regularization in surgery. We would report it as such.