Medical imaging models are often trained with synthetic data. A generative model paints realistic lesions into healthy images so the detector sees more and rarer examples. Every reported gain from this practice has been measured on curated retrospective images: sharp, expert-selected frames pulled from an archive. The images a deployed model actually receives are different. In colonoscopy the input is a compressed, motion-blurred, bubble-strewn video stream, and the polyps that matter most are the small ones. Nobody knows whether gains from synthetic lesions carry over to that stream, shrink, or reverse. There is a specific reason to worry. Generators leave their own high-frequency fingerprints in the images they produce, and live-stream degradation is itself mostly a high-frequency phenomenon.
This note proposes measuring and then fixing that gap. We run polyp detectors silently alongside live colonoscopies: the model sees the real stream, but its outputs are not shown to clinicians. We then split the drop from curated to live performance into a part caused by time passing and a part caused by the live data pipeline, using the three-way design that Mind the Performance Gap introduced for tabular hospital records. The method contribution is "deployment-conditioned synthesis". Silent-mode capture yields thousands of unlabeled live frames at essentially no cost. We train a mask-guided diffusion model (a generator that turns random noise into an image through gradual denoising) to inpaint polyps into those real live backgrounds. The result is labeled training data that looks like what the model will actually see, and no live frame has to be annotated for it.
Core Claim
Our first claim is a hypothesis to test. Generative augmentation trained on curated keyframes improves the parts of performance that retrospective benchmarks measure: overlap on clean frames and transfer across centers. It does little for the infrastructure component of the prospective gap, meaning the loss caused by live-stream degradation rather than by patients or time. It may even hurt calibration there, because the detector learns to rely on clean, generator-specific texture.
Our second claim is that one change largely restores the benefit: inserting synthetic lesions into real, unlabeled live-stream backgrounds. Two strong alternatives could match it. One is heuristic degradation augmentation. The other is fine-tuning on a modest number of labeled live frames. The experiments are designed so that either outcome is informative.
Colonoscopy is the right testbed for three reasons. It is genuinely real-time: TransNetR runs at 54.60 frames per second. That speed rules out costly per-frame generative test-time methods and pushes generation into training. Colonoscopy also has a multi-center benchmark for a retrospective analog, PolypGen with six centers. Finally, its clinical value, like that of screening mammography, depends on small lesions. In the prospective Mia trial (Prospective implementation of AI-assisted screen reading), 47.0% of the additional cancers the AI found were 10 mm or smaller.
Prior Work and What It Leaves Open
Two studies supply the evaluation method, and both stop short of imaging. Ötles et al. compared three datasets for a C. difficile risk model:
- Historical warehouse data. This is the data the model was built and tested on.
- A warehouse pull covering the deployment period. This is retrieved and cleaned the usual way, after the fact.
- The real-time feed. This is the data exactly as the deployed model received it.
The comparisons attributed a 0.016 AUROC loss to infrastructure and a +0.005 change to temporal shift, with calibration also degrading. That study involved no images and no generative modeling.
Robustness Stress Testing in Medical Image Classification motivates our choice of metrics. It found that aggregate AUC stayed stable under heuristic corruptions while true- and false-positive rates at fixed thresholds swung widely. We therefore report operating-point metrics rather than only Dice. Its corruptions were offline filters, not real deployment data.
Three generative-augmentation papers motivate and constrain the design:
- Optimizing Synthetic Data for Enhanced Pancreatic Tumor Segmentation used a mask-guided latent diffusion model to insert tumors into healthy CT. SwinUNETR improved from 45.88% to 56.10% Dice with large synthetic tumors, but only on one retrospective dataset. Its finding that boundary noise on synthetic masks drops U-Net from 48.48% to 32.49% Dice is a hard constraint on us: inpainted polyps must stay inside their masks. We will verify this with a mask-consistency check on every synthetic sample.
- Rethinking Bias in Generative Data Augmentation for Medical AI showed that generators imprint high-frequency artifacts that downstream classifiers exploit. Its frequency recalibration step (FreRec) raised DenseNet AUC on brain MRI from 0.794 to 0.855. We reuse FreRec as a competing fix, because it addresses the same suspected mechanism from the generator side rather than the background side.
- Test-time generative augmentation for medical image segmentation (TTGA) applies a domain-tuned diffusion model at inference time. It gains 0.1% to 2.3% Dice on Kvasir and the CVC polyp sets, all retrospective. Its multi-step sampling per input is incompatible with real-time video, so it serves as an offline upper reference rather than a deployable competitor.
On the shift and safety side, Automatic Dataset Shift Identification supplies a tool. It measures distance between feature distributions (maximum mean discrepancy over self-supervised embeddings), which we use to quantify how far live frames sit from the archive and from each kind of synthetic data. Avoiding Catastrophic Referral Failures shows that models become confidently wrong under covariate shift and that post-hoc logit adjustments can repair referral. That makes post-hoc recalibration a necessary competitor for the calibration part of our claim.
Research Program
The work proceeds in three parts. The silent-mode data collection runs throughout, so the timeline matters.
Part 1: Silent-mode collection. We partner with one or more endoscopy units, ideally including a vendor not represented in public training data. The detector runs silently on the live video feed, and we log every frame it sees together with its outputs. Recorded videos are later annotated by endoscopists for polyp presence, bounding masks on sampled frames, and polyp size from the procedure report.
From these recordings we build three test sets:
- D_ret: curated keyframes from the unit's historical archive, prepared the way public benchmarks are.
- D'_ret: keyframes from the silent-mode period, curated identically after the fact.
- D_pro: the uncurated live frames at the moments the model actually processed them.
The temporal component of the gap is the difference between D_ret and D'_ret. The infrastructure component is the difference between D'_ret and D_pro. Because the patients and polyps in D'_ret and D_pro are the same, this split is clean.
Part 2: Deployment-conditioned synthesis. We fine-tune a mask-conditioned latent diffusion inpainting model on public polyp data (Kvasir-SEG and PolypGen). The design borrows the mask guidance from the Difftumor-style generator used by Peng et al. We then apply it to polyp-free frames drawn from the unlabeled live capture.
The masks come from a shape model fit to real polyps. Their size distribution is deliberately skewed toward diminutive polyps, the reverse of the large-lesion preference Peng et al. found best for Dice. Mask consistency is enforced with an independent segmenter, and samples whose predicted region leaks outside the mask are rejected.
The only live data the synthesis step consumes is unlabeled background frames. No live labels are used, so the comparison against labeled fine-tuning is meaningful.
Part 3: Mechanism analysis. We test whether frequency fingerprints explain any loss. For each trained detector we measure:
- how much its predictions change when inputs are low-pass filtered;
- its feature-distribution distance to D_pro;
- its calibration on D_pro.
We then ask whether these quantities predict the size of the infrastructure-gap loss across training recipes.
Baselines
Naive reference: real data only. The detector is a TransNetR-style network trained on real public frames with standard flips and crops and no synthetic data. On Kvasir-SEG, TransNetR reports Dice 0.8706 and mean IoU 0.8016, and a 2.81% mean IoU improvement over the next-best model in PolypGen out-of-distribution testing. These numbers anchor our reimplementation. If our version falls well short of them on Kvasir-SEG, every downstream comparison is suspect.
Standard generative augmentation. The same detector is trained with polyps inpainted into curated public keyframes by the same diffusion model. This is the strongest existing approach to the question as the literature frames it. It isolates our key ingredient, because the background distribution is the only difference from our method. We found no reported result on Lacuna for diffusion-based polyp augmentation evaluated on PolypGen or on live colonoscopy, so we establish this baseline ourselves.
Standard generative augmentation plus FreRec. This applies frequency recalibration from Rethinking Bias in Generative Data Augmentation to the synthetic images. It is the strongest competitor that also targets the frequency mechanism.
Heuristic degradation augmentation. This adds motion blur, video compression, defocus, specular highlights and noise at tuned severities, with or without synthetic lesions.
Labeled-live fine-tuning. The real-only model is fine-tuned on n labeled live frames, with n swept upward. We report the n at which it matches our method.
Post-hoc recalibration. Shifted Split Referral and Split Platt Scaling from Avoiding Catastrophic Referral Failures are applied to each trained model for the calibration comparisons.
TTGA. TTGA is run offline on D_pro as a latency-unconstrained reference only.
Fairness controls. All detectors share the architecture, training steps, the number of synthetic samples (where applicable), and a fixed hyperparameter search budget. Tuning uses a held-out public validation split, never D_pro. Every method is scored at a threshold fixed on retrospective data before the live sets are touched.
What counts as success. Our method beats standard generative augmentation if it recovers a significant fraction of the infrastructure-gap loss, measured as diminutive-polyp sensitivity at a fixed false-positive rate per minute of video. Patient-level bootstrap confidence intervals must exclude zero. It must also do no worse on D'_ret, and it must match or beat heuristic degradation augmentation on the same metric.
Why not just…?
Why not just add heuristic degradations like blur and compression? This is the most serious threat. The infrastructure shift in colonoscopy may be dominated by a few degradation types that simple filters already mimic. Against it, the stress-testing study found that heuristic corruptions destabilized operating points rather than producing robustness. Filters also cannot reproduce the joint statistics of real live frames: bubbles, fluid, mucosal motion and vendor-specific processing occurring together. The heuristic baseline settles the question directly. If it matches deployment-conditioned synthesis on the D'_ret-to-D_pro sensitivity loss, we report that and the contribution reduces to the decomposition and the warning about curated-frame augmentation. That result is still useful.
Why not just label a few hundred live frames and fine-tune? This is also a real threat, since a few hundred labels may be cheap compared with building a generator. We answer it with an annotation-equivalence curve: how many labeled live frames fine-tuning needs to match our method. If the answer is small, the generative route is not worth its cost for a single site. The case for our method rests on unlabeled backgrounds being free at every new site and vendor, whereas expert labels must be gathered again each time.
Why not just recalibrate after deployment? Post-hoc logit adjustment can fix confidence and referral without retraining, as Avoiding Catastrophic Referral Failures showed. It cannot make a detector see a diminutive polyp it never responded to. We therefore apply recalibration on top of every method and evaluate detection and calibration separately. If recalibration alone closes the calibration gap, we drop calibration from our claim and keep the sensitivity claim.
Why not just use test-time generative augmentation? TTGA needs multi-step diffusion sampling for every frame. That cannot run alongside a 30-plus frames-per-second stream on endoscopy-suite hardware. Its reported gains of 0.1% to 2.3% Dice were retrospective. We run it offline on D_pro so readers can see what the latency constraint costs.
Proposed Experiments and Decisive Checks
Retrospective analog. Before live data arrives, we run leave-one-center-out training on PolypGen. Each recipe is evaluated on the held-out center and on Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, CVC-300 and ETIS-LaribPolypDB. Backgrounds for deployment-conditioned synthesis come from unlabeled frames of the held-out center. This confirms the pipeline works and gives a first estimate of whether background conditioning matters across vendors.
Live decomposition. The prospective experiment reports the full D_ret, D'_ret and D_pro decomposition for every recipe. The metrics are:
- Dice on annotated frames;
- per-polyp sensitivity at a fixed false-positive rate per minute, stratified by size (5 mm or smaller, 6 to 9 mm, 10 mm or larger);
- expected calibration error;
- referral performance under the Shifted Split Referral policy.
Three decisive checks.
- Does the hypothesized gap exist? Standard generative augmentation must show a positive gain over real-only on D'_ret that shrinks or reverses on D_pro. If the gain transfers intact, our motivating hypothesis is false. We would report that generative augmentation survives the live feed, which practitioners need to know.
- Is the background the fix? Deployment-conditioned synthesis must recover the lost gain at least as well as both FreRec and heuristic degradation, with the effect concentrated in diminutive polyps.
- Does frequency reliance explain it? Across all recipes, low-pass sensitivity and the feature distance between training data and D_pro should predict the size of the infrastructure loss. If they do not, the generator-fingerprint story is wrong even if the method works, and we say so.
Failure monitoring. Following Ötles et al.'s observation that their live feed failed completely on 10 of 356 days, we log pipeline outages and dropped frames. These are reported separately from model errors so that infrastructure failures are not misattributed to the model.
References
- Mind the Performance Gap: Examining Dataset Shift During Prospective Validation
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing
- Prospective implementation of AI-assisted screen reading to improve early detection of breast cancer
- Robustness Stress Testing in Medical Image Classification
- Optimizing Synthetic Data for Enhanced Pancreatic Tumor Segmentation
- Rethinking Bias in Generative Data Augmentation for Medical AI: a Frequency Recalibration Method
- Test-time generative augmentation for medical image segmentation
- Automatic Dataset Shift Identification to Support Safe Deployment of Medical Imaging AI
- Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift