Learning to See Ischemia from the Moment Blood Returns

When a surgeon releases the clamp on a transplanted kidney or a free tissue flap, blood flows back in. Within seconds, every perfused patch of tissue begins to flicker in color, very slightly, in time with the heartbeat. Tissue that will die does not flicker. Remote photoplethysmography (rPPG) recovers this flicker from ordinary video; it estimates the pulse signal from tiny skin-color changes, without any contact sensor. Surgical studies already show that this local pulse separates viable from non-viable tissue. What they cannot do is draw the boundary automatically. Training a network to draw it would normally require surgeons to trace perfusion boundaries by hand, and those boundaries are often invisible to the eye.

We propose to get pixel-level training labels from a single timestamp that the operating room already records: the moment of clamp release, or more generally any known perfusion event. Tissue whose local pulse is absent before the event and present after it is the reperfused territory. Tissue that pulses throughout is surrounding healthy tissue. Instruments, gauze and dead tissue never pulse. We would train a segmentation network on these event-anchored pseudo-labels, then deploy it on short clips that contain no event at all, such as later intraoperative checks or postoperative monitoring. Its job there is to output a perfusion mask that holds up under tissue deformation and camera motion. In two sentences: one routine clinical timestamp, combined with before/after pulse contrast, turns hours of unannotated surgical video into dense segmentation supervision. A network trained on that supervision should produce perfusion masks from a few seconds of video, where physics-only maps need long, stable windows and a human to read them.

Core Claim

Each of the two directions supplies a key ingredient. Spatial-temporal physiological mapping supplies the measurement. Local signals from small patches are stacked over time, so that the pulse shows up as synchronized periodic structure. The annotation-bottleneck literature supplies the strategy: replace dense masks with routine clinical metadata. This mirrors how Weakly Supervised Lesion Co-Segmentation on CT Scans turned radiologists' routine two-line tumor measurements into 90.3% Dice masks. It also mirrors how Domain Adaptation for Ulcerative Colitis Severity Estimation used patient-level diagnoses in place of frame-level grades.

The new claim is that a perfusion event timestamp plays the role of the RECIST line for physiological video, and that its value comes from contrast. Labeling patches by pulse strength in a single state is confounded by motion, specular glare and illumination drift, all of which affect perfusion-independent pulse estimates. Comparing the same tissue across an event cancels much of that confound, because glare, texture and skin tone stay roughly constant while perfusion changes.

The motivating evidence is clinical. Perfusion assessment via local rPPG computes multi-scale local pulse maps with the Plane-Orthogonal-to-Skin projection (POS, a fixed color-space projection that isolates pulse) and a correlation $\rho$ against reference skin. It flagged necrosis five weeks before it became clinically visible, but its output is a heatmap that a person interprets, or summary features fed to a support vector machine. Continuous intraoperative perfusion monitoring of free flaps used manually placed static regions on 60 fps microscope video. In that study, flap correlation rose from near zero to 0.56 after reperfusion, detection took on average 18 seconds after clamp release, and ischemic and perfused states differed with $p<0.001$. Both papers establish that the signal exists and that the event is sharp. Neither learns anything, and neither segments. That gap is what we target.

Research Program

The first component is label generation. We track tissue with dense optical flow so that each patch $i$ follows the same physical tissue as it deforms. This replaces the rigid face grids of RhythmNet and the landmark-based maps of Joint Spatial-Temporal Modeling and Contrastive Learning, which assume a face. For each tracked patch we compute a local POS pulse. We then compute its spectral coherence with a reference region, $c_i^{-}$ in a window before the event and $c_i^{+}$ in a window after it. The soft pseudo-label is

$$ y_i = \sigma\big(\alpha\,(c_i^{+} - c_i^{-})\big)\cdot \sigma\big(\alpha\,(\tau - c_i^{-})\big), $$

where $\sigma$ is the logistic function, $\alpha$ sets sharpness and $\tau$ is a coherence level below which tissue counts as non-pulsing. Under this rule a patch is labeled "reperfused" only if it was silent before and pulsing after. A second, analogous label marks "always perfused" tissue, where $c_i^{-}$ and $c_i^{+}$ are both high. Everything else is "non-perfused."

The second component is gating. Pseudo-labels are admitted through two gates. The first is a temporal gate in the style of the curriculum signal-to-noise filtering of Semi-rPPG, which admits the cleanest 20% of pulse estimates first and grows to 80%. The second is a spatial gate based on Monte Carlo dropout uncertainty, following Learning to Segment Medical Images with Scribble-Supervision Alone. This combination is how we address the open tension between temporal and spatial filtering. We use the two gates jointly because the file reports that each alone was needed to stop confirmation bias in its own setting: SNR for 1D pulse in Semi-rPPG, spatial uncertainty for masks in Can et al.

The third component is the student network. It sees only a short single-state window of 2 to 6 seconds and no event. Its input is the tracked spatial-temporal map: patches along one axis, time along the other, in a YUV color representation, which RhythmNet found better than RGB. It outputs per-patch perfusion probabilities, which are interpolated back to a pixel mask. We would initialize from an encoder pretrained without labels on face video, because UP-Phys found that its spatial attention module amplified noise when trained from scratch but converged in one epoch from pretrained weights. We also add the label-free frequency losses of Qian et al. (bandwidth, sparsity, periodicity) and the time-reversal consistency of Semi-rPPG as auxiliary terms. On the backbone, Semi-rPPG reports that Transformers were unstable in semi-supervised rPPG while 3D CNNs were stable, whereas Qian et al. trained a spatial-temporal Transformer on precomputed maps successfully. We treat this as a constraint, not a choice already made. We will compare a 2D CNN with a GRU on maps against a 3D CNN on raw clips, and keep whichever is stable.

Baselines

The naive reference point is the Kossack et al. local POS correlation map, binarized with an Otsu threshold and computed over the same short window the student sees. It needs no training and is exactly what a surgeon would get today. To keep it fair we give it the same optical-flow tracking as our pipeline, and we tune its window, pyramid scale and threshold on the same validation clips. Beating it means higher Dice against expert masks at equal window length, with the margin growing under camera motion.

The strongest existing competitor is the same physics map computed over a long, event-spanning window. In other words, it is our own label generator used directly as the segmentation, with no student. We treat this as the upper bound on what the labels contain. For context, Schraven et al. report detection of reperfusion within an average of 18 seconds and a flap correlation of 0.56 after reperfusion, and Kossack et al. report 96.8% accuracy for real-versus-silicone-mask detection. We found no reported result on Lacuna for pixel-level perfusion segmentation Dice on any surgical dataset, so the first numbers will come from this study. The student is not expected to beat its teacher on event-spanning clips. Its win condition is to come within a few Dice points of the teacher while using only short, event-free windows, where the teacher cannot run at all.

The annotation baseline is the Can et al. scribble pipeline: random-walk seeding, a dense conditional random field (a classical boundary-smoothing step) and uncertainty masking. It is trained on scribbles that surgeons draw on a subset of training frames under the same compute and tuning budget. This baseline prices our approach in annotation time. Matching it at zero drawing cost counts as success. The semi-supervised option of Multi-Constraint Collaborative Self-Training for Cardiac MRI is run as a second annotation baseline with 5% and 10% of frames masked. On cardiac MRI it reached 0.730 and 0.801 Dice at those fractions, against 0.891 with full supervision.

Three ablations isolate the key ingredient. The first removes the event and labels patches by single-state pulse SNR alone. This is the spatial analogue of the rPPG-Toolbox weak mode, in which rPPG-Toolbox supervises networks with POS outputs. The second removes each gate in turn. The third replaces tracked patches with a fixed grid.

Why not just…?

Why not just threshold the POS correlation map and skip learning? This is the real threat, and we take it seriously. On slow, stable microscope footage with long windows, Schraven et al. show that physics alone separates the two states cleanly. The case for learning rests on two things. The first is short windows: spectral coherence over two seconds is noisy, and a network can draw on spatial context. The second is motion: the file reports that handcrafted landmark pipelines rise to 24.17 BPM error under head rotation on SYNC-rPPG, against 11.95 BPM for learned weighting in SkinMap. Both claims come from face data. The window-length sweep below settles the question for tissue. If the tracked, thresholded map matches the student at 2 to 6 second windows, we will report that the contribution is the event-anchored labeling and evaluation protocol, not the network.

Why not just train on single-state POS pseudo-labels, as the rPPG-Toolbox weak mode does? That would be the simplest learned alternative. Its weakness is that low single-state pulse is ambiguous between ischemia, glare, occlusion and motion blur. Our event-free ablation measures exactly this. We predict it confuses specular highlights and smoke with ischemia, the kind of spurious-cue failure SpurCon documents for weakly supervised medical models. If it matches the event-anchored version, the event adds nothing and we drop it.

Why not just have surgeons scribble, or segment by RGB appearance? Scribbles work well on static anatomy, with 88.0% against 91.2% Dice on cardiac MRI in Can et al. Perfusion boundaries are a different case: early ischemic tissue can look normal, which is why Kossack et al. could anticipate necrosis weeks before it was visible. Appearance-driven refiners such as GrabCut have no color contrast to work with. Agarwal et al. also found that dense CRF refinement cut recall sharply. The scribble baseline is included so this argument is tested empirically and not assumed.

Proposed Experiments

The first decisive experiment uses physically known ground truth and needs no clinical access. We would record forearm and hand video of volunteers during blood-pressure-cuff occlusion and release, with a finger contact PPG confirming the ischemic interval. Once the cuff is inflated, the territory downstream of it is anatomically defined. A non-expert can therefore mark the hand once per subject, and that mask is the ground-truth perfused region during release. We vary skin tone, hand motion and lighting. Following the concerns rPPG-Toolbox raises about demographic bias, we will stratify results by skin type. The metrics are Dice and boundary distance against the hand mask, curves of Dice against window length, and time from cuff release to a stable mask. That last number is directly comparable in kind to the 18-second reperfusion detection of Schraven et al.

The second experiment uses clinical surgical video, through collaboration with the groups behind SurgTissue and the free-flap microscope dataset. Clamp-release times are logged intraoperatively, and surgeons draw boundary masks on a small held-out test set only, after the outcome is known. The same baselines and metrics apply. We add one clinical check: whether the student's non-perfused fraction on the single failed flap and on the necrosis case separates them from successful cases.

The third experiment is a sanity transfer back to faces. We evaluate whether the student's skin mask, used as a pulse-extraction weighting, preserves heart-rate accuracy on UBFC-rPPG and PURE relative to the reported UP-Phys errors (MAE 0.18 and 0.48 respectively). This checks that the masks capture pulse-bearing tissue and not texture.

Three results would falsify the central claim. The first is the event-free ablation matching the event-anchored model. The second is the tracked, thresholded POS map matching the student at short windows. The third is cuff-experiment Dice that collapses under hand motion even with tracking. Any one of these would tell us where the value of the idea actually lies.