Resolvent Refinement: Edge-Aware Shift-Invert Filtering for Real-Time Medical Segmentation

Fast medical segmentation networks get their speed by throwing away fine detail. S2M-Net, for example, mixes information across the whole image with a Fourier transform but keeps only a central 32×32 block of low frequencies. That choice gives 10 ms latency, and its authors acknowledge that it discards boundary information. The detail being lost is exactly what matters most when anatomy varies from patient to patient: thin vessels, irregular tumor margins, and the jaws of articulated instruments. This note proposes a different way to suppress high frequencies.

Instead of cutting off spatial frequencies, the network applies the inverse of a shifted graph Laplacian. A graph Laplacian is an operator that measures how much a signal changes between neighbouring pixels. Here its edge weights are learned from the current frame, so weights are small across image boundaries. Applying the shifted inverse, $(L+\sigma I)^{-1}$, damps oscillations inside a region while leaving changes across real boundaries largely intact. Each application is one sparse linear solve. It can be done approximately with a few conjugate-gradient iterations, and in video it can be warm-started from the previous frame's solution. The result is a small refinement module that recovers boundary detail at a cost of a few milliseconds.

Core Claim

The shift-invert transformation — replacing an operator $A$ by $(A-\sigma I)^{-1}$ so that its unbounded high end is squeezed toward zero — was developed in operator learning to make data-driven spectral methods stable. We claim it also makes a good output layer for real-time segmentation once $A$ is an image-adapted Laplacian rather than a Fourier multiplier. Concretely, a coarse fast network produces low-resolution logits $b$, and the refined logits are

$$ u \;=\; \sum_{j=1}^{J} c_j \,(L_\theta(x) + \sigma_j I)^{-1}\, b , $$

where $L_\theta(x)$ is the Laplacian of a pixel grid whose edge weights are computed from frame $x$ by a learned feature map with parameters $\theta$. The $\sigma_j>0$ are shifts, and the $c_j$ are mixing coefficients that may be negative. The shifts and coefficients are predicted per image from the coarse prediction.

A single resolvent acts as an edge-aware low-pass filter whose reach is about $1/\sqrt{\sigma}$ pixels. A difference of two resolvents acts as a band-pass filter on graph frequencies, which lets the module pick a structure scale per image. That per-image scale is how the module responds to morphological variability.

Because $L_\theta$ is positive semidefinite and degree-bounded on an 8-neighbour grid, $L_\theta+\sigma I$ is symmetric positive definite with condition number at most $(16+\sigma)/\sigma$. Conjugate gradient therefore converges in $O(\sqrt{16/\sigma})$ iterations from a cold start, and needs fewer from a warm one. The inexact solves are justified by the same argument used in Faster Eigenvector Computation via Shift-and-Invert Preconditioning. Gradients pass through the solve by one additional solve with the same symmetric matrix.

The novel element is not edge-aware smoothing itself. It is the combination of three things. First, learned, morphology-conditioned multi-shift resolvents serve as the replacement for hard frequency truncation in a real-time segmenter. Second, the module is trained end to end. Third, warm-started streaming solves are run whose initial residual doubles as a principled frame-continuity signal.

Where this sits in prior work

The operator-theory papers supply the mechanism and its justification. Krylov subspace methods for estimating operator-vector multiplications in Hilbert spaces shows that projecting an unbounded operator directly can stall or diverge, while shift-invert projections converge, exponentially under a holomorphy condition. Krylov Subspace Method for Nonlinear Dynamical Systems with Random Noise makes the same move for noisy transfer operators. Laplace Transform Based Low-Complexity Learning of Continuous Markov Semigroups estimates the resolvent of a generator at linear cost. Two related papers, On Occupation Kernels, Liouville Operators, and Dynamic Mode Decomposition and Dynamic Mode Decomposition for Continuous Time Systems with the Liouville Operator, bound the operator instead by a scaling factor. All of these work on dynamical systems such as molecular trajectories, flows and EEG. None of them touches images. We borrow the stabilizing transformation and apply it to a spatial operator built per frame.

Two papers set the inexact-solve and shift-selection constraints. Faster Eigenvector Computation via Shift-and-Invert Preconditioning proves that shifted systems need only approximate solutions and that streaming refinement works given a warm start. Its weakness, in the file's reading, is that the warm start must come from elsewhere and the data must be independent and identically distributed. Video frames are neither independent nor identically distributed. They do, however, supply a natural warm start: the previous frame. The file also notes, citing Hashimoto et al. 2021, that choosing the shift without knowing the spectrum is unsolved. We do not solve it in general. We sidestep it by learning a per-image shift predictor supervised only through segmentation loss, and we test whether that beats a fixed shift.

Two papers motivate the gap from the segmentation side. S2M-Net is the closest prior work. It uses fixed rectangular Fourier truncation for speed plus a morphology-aware loss that computes tubularity, compactness, irregularity and scale descriptors. We keep its fast backbone idea and replace the truncation's consequence, lost boundaries, with an operator that is adapted to each image. FAMNet shows that anatomy concentrates in mid-frequency bands. That result motivates the band-pass, difference-of-resolvents form, but FAMNet's bands are Fourier bands fixed across images, whereas ours are graph bands that bend around each image's edges.

Two spectral-geometry papers supply precedent for steering operators toward morphology. Multi-atlas Spectral PatchMatch shows that Laplacian spectral coordinates resist intensity and registration errors, keeping myocardium Dice at 0.801 versus 0.800 under crude affine registration. It needs offline eigensolvers. We never compute eigenvectors; we only apply resolvents, which is what makes real-time execution feasible. Hamiltonian Operator for Spectral Shape Analysis adds a potential to the Laplacian to localize its basis. A spatially varying shift $\sigma(x)$ is exactly such a potential, and we treat it as an optional extension.

The video papers motivate the streaming component. VPSentry and the DMNet paper, Efficient Global-Local Memory for Real-Time Instrument Segmentation of Robotic Surgical Video, gate their memories with heuristic correlation and entropy thresholds. The relative residual of a warm-started solve, $\|b_t - (L_t+\sigma I)u_{t-1}\|/\|b_t\|$, measures how badly the previous frame explains the current one, and it comes free with the solver.

Research Program

We will build the module on two fast backbones: a MobileNetV2 encoder with a contracting decoder in the style of M2U-Net, and a reimplementation of S2M-Net's truncated spectral-mixing network. The coarse logits are upsampled bilinearly to full resolution and passed into the resolvent layer.

The edge weights take the form $w_{ik}=\exp(-\|g_\theta(x)_i-g_\theta(x)_k\|^2)$ on an 8-neighbour grid. The feature map $g_\theta$ consists of two shallow convolutions on the image. A tiny head reads pooled coarse features and outputs $J\in\{1,2,3\}$ shifts on a log scale together with their coefficients. Optionally, an auxiliary loss ties that head to S2M-Net-style scale and tubularity descriptors of the ground-truth mask.

Solves use a fixed budget of $K$ conjugate-gradient iterations implemented as fused stencil kernels. Training uses the implicit gradient. We will measure end-to-end latency on one GPU class at 352×352 and 768×768 to match S2M-Net's reporting, and on an embedded ARM board to match M2U-Net's.

The analysis has three parts. First, we will characterize the learned filters by applying the layer to synthetic edge and line patterns. Second, we will relate predicted shifts to lesion size and tubularity on held-out data. Third, we will measure how solve accuracy (the iteration budget $K$) trades against Dice and boundary error.

For video, we will report iterations-to-tolerance with cold versus warm starts. We will also test whether the warm-start residual detects the frames VPSentry's gate would reject.

Baselines

The naive reference is the coarse backbone with bilinear upsampling and no refinement. This is the "do nothing" point that tells us how much boundary detail is actually missing.

The strongest existing method on the image side is S2M-Net. It is the only cited model that jointly targets latency and morphological diversity. The file reports 83.43% Dice on EndoVis 2017 for it, with 10.1 ms latency at 352×352 and 4.7M parameters. We will retrain our reimplementation under our splits and report both our reproduced number and theirs, then add our layer on top. Beating it means higher Dice and lower boundary error at an added latency of at most 3 ms, with gains concentrated in the thin and irregular strata.

On retinal vessels, the reference is M2U-Net. The file reports Dice 0.8091 on DRIVE, 0.8006 on CHASE_DB1, and 0.7814 on HRF, at 19.9 ms on HRF on a GTX 1080 Ti. Our module on the same backbone must improve Dice on HRF without exceeding real-time on that hardware class.

On video, VPSentry is the strongest semi-supervised polyp baseline. The file reports 83.95% Dice on SUN-SEG-Hard with 10% labels at 28–30 FPS on an RTX 4090. PNS+, from Video Polyp Segmentation: A Deep Learning Perspective, reports 0.756 Dice on SUN-SEG-Easy unseen at 170 FPS and is the speed reference. On EndoVis 2017 instrument-type segmentation, DMNet reports 61.03% mean Dice at 38 FPS.

For boundary metrics such as boundary F-score or 95th-percentile Hausdorff distance, we found no reported result on Lacuna for any of these baselines. We will compute them ourselves for every model.

Every comparison uses identical training data and splits, the same backbone weights where shared, and the same tuning budget, with an equal number of hyperparameter trials per method. Latency is measured on the same device at the same batch size.

Two ablations isolate the key ingredient. The first is a single learned global shift, which is close to classical edge-aware smoothing. The second is the same module with the Laplacian replaced by the plain, non-edge-aware grid Laplacian. That replacement makes the resolvent a smooth Fourier filter and separates "smooth rather than hard cutoff" from "image-adapted basis."

Why not just…?

Why not just add a heavier decoder with skip connections? Restoring detail through high-resolution skip features is the standard answer. M2U-Net shows that a light decoder already performs well on retinal vessels. This alternative is a real threat. We will settle it by giving the decoder the exact parameter and latency budget the resolvent layer consumes, and comparing boundary error on the thin and irregular strata. If the decoder matches us, the operator view adds nothing practical and we will say so.

Why not just use a learned smooth Fourier filter instead of hard truncation? A global per-frequency filter removes the abrupt cutoff for the cost of one FFT. However, any Fourier filter treats an edge and texture of the same frequency identically. FAMNet's mid-band finding concerns Fourier bands that are fixed across images. The plain-grid-Laplacian ablation is precisely a smooth isotropic Fourier-domain filter. If it matches the edge-aware version, the claim that the image-adapted basis matters is refuted.

Why not just post-process with an edge-aware smoother such as a guided filter or a CRF? A fixed-strength edge-aware smoother amounts to a single resolvent with one global shift. That is our first ablation, run with the same learned edges. If multiple, morphology-predicted shifts do not beat it on small or irregular structures, our contribution reduces to end-to-end training of a known operation. We would report that outcome rather than dress it up.

Why not just use a heuristic memory gate for video? VPSentry's gate already works at 28–30 FPS. Our claim is narrower: the solver residual is an equally good continuity signal that costs nothing extra. We will test it by measuring how well the residual predicts VPSentry-rejected frames and annotated scene changes.

Proposed Experiments

The first decisive check runs on EndoVis 2017, DRIVE, CHASE_DB1 and HRF. It compares coarse-only, S2M-Net-style truncation, truncation plus our layer, and the ablations, at matched latency. Results are stratified by a per-sample tubularity and size descriptor. The prediction is that gains come from boundary error on thin and irregular samples, with near-zero change on compact ones. A uniform small gain would instead suggest generic smoothing.

The second check is the shift study. We correlate predicted $\sigma_j$ with lesion scale, and compare learned per-image shifts against the best fixed shift chosen on validation data. This directly tests whether adaptive shift selection — an open problem in the operator literature — has a usable learned answer in this setting.

The third check is a slice-wise stress test on MSD-Pancreas tumors. DMC-Net reports 60.71% tumor Dice there, against roughly 52% for a 3D U-Net and 54% for nnU-Net. Because our model is 2D, we will report this only as a probe of irregular-boundary behaviour, not as a head-to-head comparison.

The fourth check uses SUN-SEG and the EndoVis video sequences. We measure CG iterations and latency with cold and warm starts, and verify that 30 FPS holds on the target GPU. We report how well the warm-start residual flags discontinuous frames.

The proposal fails cleanly in two cases. One is if the plain-grid or single-shift ablation matches the full module. The other is if added latency exceeds about 3 ms at 352×352.