Endoscopy images of the colon are full of glare. Wet tissue reflects the scope's light straight back into the camera, producing bright white blobs that wipe out the tissue texture underneath and create sharp edges that look like lesion borders. Now add the usual clinical constraint: a team has a few hundred expert-annotated frames, all from one hospital, one endoscope model, and one light source. A segmentation network trained on that data can learn that particular hospital's glare: its color, its size, its typical placement. It will then fail in two ways. It will flag glare as polyps, and it will break when moved to a hospital whose light and sensor produce different glare.
This note proposes a cheap, physics-based fix that needs no extra labels. We build "counterfactual glare" versions of each frame. In one version the highlight is suppressed. In others it is recolored, rescaled, or moved, using the standard physical model of how light reflects off a surface. We then train the network to give the same answer on every version, everywhere except the pixels the glare has truly destroyed. Because this consistency signal needs no annotation, it uses the large pool of unlabeled frames that every single-center team already has. The claim is that, under few labels from a single center, much of the false-positive rate and much of the cross-center accuracy drop come from the model relying on glare. We further claim that removing that reliance works better than either erasing glare or adding generic augmentation.
Core Claim
Our hypothesis is that glare is a center-specific confound. Small-sample training is especially prone to latching onto it, much as the chest X-ray investigation found models learning the operational signatures of how data was collected rather than the disease. We make two testable predictions. First, a simple "glare reliance" score predicts how much a model loses when evaluated at a new hospital. The score is the fraction of predictions that change when only the glare changes. Second, enforcing invariance to physically plausible glare changes on unlabeled frames from the training hospital lowers that score, and lowering it improves accuracy at hospitals never seen in training. The second prediction is the new part. Prior work either removes highlights, synthesizes them for augmentation, or designs architectures that work around them. None of it measures glare reliance as the cause of cross-center failure, and none of it trains against that reliance directly while protecting saturated pixels from hallucinated content.
Research Program
The first component estimates glare physically. Under the dichromatic reflection model, each pixel is a mix of a diffuse part, which carries the tissue color, and a specular part, which carries the light-source color. We separate the two with the maximum-chromaticity analysis of Separating Reflection Components Based on Chromaticity and Noise Analysis. We reuse that paper's method, including its camera-noise model. This matters for our setting because the paper shows that sensor dark noise mimics specularity in chromaticity space, and different hospitals use different sensors. Without the noise correction, the glare estimate would itself carry a center signature. The output for each frame is a soft specular map, an estimated light chromaticity, and a binary map of saturated pixels where the tissue information is unrecoverable, as described for clipped highlights in An Image Highlights Removal Method with Polarization Principle. We deliberately avoid capture hardware such as rotating polarizers. That paper's setup is incompatible with flexible endoscopes, and our goal is a software-only method.
The second component generates counterfactual frames. Given the decomposition, we produce variants of each frame. One attenuates the specular component wherever it is not saturated. One recolors it with a light chromaticity drawn from a broad physical prior instead of the training center's own light. One rescales the specular intensity and blob size. One transplants highlight maps taken from other frames of the same video, which mimics the viewpoint-dependent movement of glare described in Rectified Surface Mosaics. We never paint invented tissue into saturated pixels.
The third component is the training objective. For a segmentation network $f$, we combine the usual supervised loss on the few labeled frames with a consistency term on unlabeled frames:
$$\mathcal{L} = \mathcal{L}_{\text{sup}} + \lambda \, \mathbb{E}_{x, T}\big[\, \lVert M_x \odot (f(x) - f(T(x))) \rVert_1 \big],$$
The symbols are as follows. $x$ is an unlabeled frame. $T$ is a randomly drawn glare counterfactual. $M_x$ is a mask that is zero on pixels saturated in either $x$ or $T(x)$ and one elsewhere. $\odot$ is pixelwise multiplication. $\lambda$ weights the term. Inside the saturated region, the network is trained to output low confidence rather than a guess. The diagnostic glare-reliance score, defined above as the share of predictions that change when only the glare changes, is measured on held-out hospitals and is never used for training.
Baselines
The naive reference point is the same segmentation backbone trained only on the labeled single-center frames with standard flips, crops, and color jitter. Every method shares this backbone, the labeled and unlabeled pools, the training budget, and an equal hyperparameter search of 20 configurations, selected on a validation split from the training center only. Choosing settings on the held-out hospitals would leak the target.
The strongest semi-supervised competitor is VPSentry, which segments polyps in video from partial labels using a prototype memory, and stops memory updates on corrupted or glare-saturated frames. It is the closest match to our regime because it already uses unlabeled frames and already treats glare as a problem. It handles glare by skipping bad frames, whereas we train invariance into every frame. The strongest architectural competitor is HFSTI-Net, which mixes frequency-domain and spatial features so that global spectral cues can bypass local glare. We run both with their published designs on our splits. We also add our consistency term on top of each one, to test whether the gains add up. No reported result was found on Lacuna for VPSentry or HFSTI-Net on single-center-trained, held-out-center polyp segmentation. The file gives no polyp segmentation numbers at all, so all comparisons will be numbers we produce ourselves. Beating these methods means a higher mean Dice score on held-out hospitals, averaged over at least three seeds, with the confidence interval excluding zero. It also means a lower glare-coincident false-positive rate, defined as false positives that overlap the specular map.
Two ablations isolate the key ingredient. The first removes the saturation mask $M_x$, so the network is pushed to agree even where content was destroyed. If the masking matters, this variant should hallucinate boundaries inside glare. The second replaces the physical counterfactuals with random bright ellipses pasted onto frames. This tests whether the physics, meaning light chromaticity, noise-aware decomposition, and realistic placement, matters or whether any bright blob would do. The glare decomposition itself is checked on the highlight benchmarks in the file. On SHIQ, Neural Solver of Dichromatic Reflection Model for Specular Highlight Removal reports PSNR 34.501 dB and SSIM 0.979. On PSD, DHAN-SHR reports PSNR 25.28 dB. We do not need to beat these. They only bound how good the glare estimate could be, and the experiments below test whether a better estimate matters downstream.
Why not just…?
Why not just remove the glare with a pretrained highlight remover before training? This is a real competitor, so we run it directly. Every frame is passed through the neural dichromatic solver, or through DHAN-SHR as the faster option, before training and testing. We expect two weaknesses. First, removers fill saturated regions with invented texture, which the HFSTI-Net authors and others note as a continuing source of false detections. Second, the remover is itself trained on non-endoscopic data and adds its own domain shift and latency. If removal matches our method on held-out-hospital Dice and glare-coincident false positives, the simpler pipeline wins and we will report that. In that case, the useful result would be the glare-reliance diagnostic.
Why not just use the existing glare augmentation from CLTS-GAN? CLTS-GAN learns to render color, lighting, texture, and glare onto virtual colonoscopy frames. It is the closest prior idea. It differs from ours in that it generates new images for augmentation rather than making paired counterfactuals of real frames, and it has no notion of protected saturated pixels. We include it as a matched-budget augmentation baseline, both with and without our consistency loss. If CLTS-GAN augmentation alone closes the cross-center gap, then pairing and masking add nothing, and the random-ellipse ablation would likely show the same thing.
Why not just freeze a large pretrained encoder and train a small head, which already resists overfitting on small cohorts? This recipe works for scarce clinical speech data, as in Shafran's frozen speech model with a Perceiver head. Strong generic features may already ignore glare. We test this with a frozen general-purpose vision encoder plus a light segmentation head, and we measure its glare-reliance score. If that score is already low and held-out accuracy is already high, our method is unnecessary for such encoders. We would then report the gain, if any, from adding the consistency term only to the small head.
Proposed Experiments
The main experiment trains on one hospital and tests on the others. We use public colonoscopy polyp collections that record which center each image came from. The file names no such benchmark, so the specific collection is chosen at execution time, with each held-out hospital reported separately. The labeled budget is varied at 50, 200, and 1000 frames, with all remaining training-center frames left unlabeled. Metrics are Dice and intersection-over-union on held-out hospitals, false positives per frame, glare-coincident false positives, and the glare-reliance score. A smaller-scale check on industrial surfaces with glare, using the photometric-stereo defect set from A dataset for surface defect detection on complex structured parts based on photometric stereo, tests whether the mechanism holds outside medicine.
Three checks decide the outcome. The first tests the cause. Across all trained models, including baselines, seeds, and labeled budgets, we correlate glare-reliance on held-out hospitals with the size of the accuracy drop from the training center. A weak correlation would mean glare is not a major driver of center shift, and the central hypothesis fails. The second tests the remedy. Our method should lower glare-reliance and raise held-out Dice compared with VPSentry, HFSTI-Net, pre-removal, and CLTS-GAN augmentation. The gain should be largest at 50 labels and shrink as labels grow, which is the pattern expected if the confound matters most when data is scarce. The third tests the physics. The full method should beat both the random-ellipse ablation and the variant without noise correction, and the gap should be larger at hospitals whose sensors differ most from the training hospital's. If none of these gaps appears, the physical modeling is unnecessary and a simpler glare augmentation should be preferred.
References
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Separating Reflection Components Based on Chromaticity and Noise Analysis
- An Image Highlights Removal Method with Polarization Principle
- Rectified Surface Mosaics
- VPSentry: Semi-supervised Video Polyp Segmentation via Sentry-guided Long-term Prototype Fusion with Correlation Dynamic Propagation
- HFSTI-Net: Hierarchical Frequency-spatial-temporal Interactions for Video Polyp Segmentation
- Neural Solver of Dichromatic Reflection Model for Specular Highlight Removal
- Dual-Hybrid Attention Network for Specular Highlight Removal
- CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy
- A dataset for surface defect detection on complex structured parts based on photometric stereo