Teaching Capsule-Endoscopy Lesion Detectors to Distrust Bubbles: Reliability-Gated Semi-Markov Training

A capsule endoscopy exam is a camera pill that records several hours of video of the digestive tract. Doctors want a list of the brief episodes where a rare lesion is visible, such as active bleeding, an ulcer, or a vascular lesion. Two things make this hard. Lesions appear in a tiny fraction of frames, and long runs of frames are degraded by bubbles, debris and motion blur. Today's best system, VISTA, scores frames with frozen foundation models and an ensemble of imbalance-aware heads. It then cleans the timeline afterward with an anatomical ordering rule, morphological opening and closing (fixed-width filters that fill short gaps and delete short blips), and per-label thresholds. That cleanup step carries a large share of the result: removing it drops validation temporal [email protected] from 0.4730 to 0.3002. It is also the crudest part of the system. Fixed-width filters cannot tell a three-frame bleeding event from a three-frame specular glint. Debris that sticks around for twenty frames gets through any window short enough to keep real short events.

We propose to replace the hand-set filters with a learned segment model and to train the frame scorer through that model. The segment model is a hidden semi-Markov model, meaning a state model that explicitly scores how long each event and each background stretch lasts. Its per-frame evidence is weighted by a learned reliability score that says how far each frame's visual evidence should be trusted. Reliability is supervised directly by the bubble and dirt annotations that the Galar dataset already provides for every frame. The spatial evidence comes from a light multi-scale attention head that pools frozen DINOv3 patch tokens at several window sizes, so that small erosions are not averaged away in a global image token. In two sentences: let the detector learn, from labelled artifacts, when its own frames are untrustworthy, and train it to produce good segments rather than good frames. We expect this to fix both failure modes of fixed filters, which are persistent artifacts producing false events and short real events being erased.

Core Claim

Our claim concerns where the remaining errors in long rare-event video come from. They are not mainly a lack of spatial features. They come from frame confidences whose errors are correlated in time and tied to visible artifacts. The fix is to model that correlation explicitly. For one pathology, a candidate segment from frame $a$ to frame $b$ receives the score

$$ S(a,b,y) = \sum_{t=a}^{b} r_t \, \ell_t(y) + \log p_y(b-a+1 \mid z), $$

where $y$ is either "event" or "background", $\ell_t(y)$ is the frame-level log-score from the multi-scale head, and $r_t \in [0,1]$ is the predicted reliability of frame $t$. The term $p_y(\cdot \mid z)$ is a learned duration distribution for state $y$, conditioned on the current anatomical region $z$. That region is decoded with the same one-way mouth-to-colon ordering that VISTA uses.

Training maximizes the conditional likelihood of the annotated segmentation, computed exactly by the semi-Markov forward recursion. A binary cross-entropy term on the technical labels is added for $r_t$. Inference is full-sequence Viterbi decoding, the dynamic-programming search for the single best segmentation. Capsule exams are reviewed offline, so the real-time latency constraint that drove the purely post-hoc HMM in the RAPN surgical phase work does not apply here.

We also expect a second benefit. The loss is defined over segments, so the hours of background contribute a handful of long segments rather than millions of frames. We expect this to reduce the swamping of the rare class by the common class that leaves a plain ResNet-50 at F1 0.14 on blood in Galar despite 98% accuracy. This is a hypothesis to test, not an assumption.

Relation to Prior Work

Three groups of papers play different roles here.

The first group motivates the problem. VISTA shows that temporal decoding matters as much as the backbone, but its decoding is a separate stage built from binary morphology on frozen features. The file also lists, as open issues, its heuristic windows and validation-tuned fusion weights. The RAPN pipeline of Swin-Tiny plus HMM Viterbi gains an order of magnitude in segmental edit score from decoding. It too trains the visual model unaware of the decoder, and it reports instability exactly when instruments or the camera move, which is the artifact-driven regime we target.

The second group supplies the method. Classifier-HMM hybrids from Smyth et al. and Hayashi et al. establish that a transition or duration prior can demand sustained evidence before raising an alert. We keep that idea, but we learn durations and we make the "evidence" itself reliability-weighted. The multi-dilation attention modules in AEDNet and DPSDA-Net show that multi-scale gating preserves tiny low-contrast targets in single images. Our head borrows the idea of windows at several scales, but applies it as cheap pooling over frozen tokens rather than as a heavy convolutional trunk. This sidesteps the compute cost both papers acknowledge.

The third group constrains the design. Temporal Label Smoothing softens labels in time and beats focal loss and survival models for early prediction. However, it assumes the evidence decays monotonically toward an event, and a capsule lesion is not a forecast target with a lead time. We therefore model segment durations rather than soften onsets. SIRST-5K shows that teaching a detector about hard negatives cuts false alarms sharply in static images. Galar's real, temporally persistent artifact labels let us do the temporal analogue without synthesizing clutter.

Research Program

All methods share the same frozen DINOv3 ViT-L/16 patch tokens, and optionally EndoFM-LV clip features, to match VISTA's inputs. On top of these we build four things.

The first is a multi-scale attention pooling head. It applies learned attention over the patch grid within windows of 1×1, 3×3 and full-grid extent, then gates the three pooled vectors per frame. A small temporal convolution with a receptive field of a few frames maps the gated features to per-pathology scores $\ell_t$.

The second is a reliability head that predicts $r_t$ from the same features. It is supervised by Galar's bubble, dirt and other technical labels, then mapped to a scalar reliability.

The third is a semi-Markov layer with one binary event chain per pathology and a shared anatomical chain. Durations are parameterized as discretized negative-binomial distributions per pathology and region, with maximum segment length capped for tractability. Long background stretches are handled by allowing background segments to chain.

The fourth is end-to-end training of the head, the reliability weighting and the durations through the forward-algorithm likelihood.

We will also analyze what the model learns. We will check whether learned reliability falls on unlabelled degraded frames, such as blur, and not only on annotated bubbles. We will check whether the learned duration distributions differ across lesion types in clinically sensible ways. We will measure how often decoded events begin inside artifact runs.

Baselines

The naive reference is the frame classifier with per-label thresholds and no temporal processing. It uses the same frozen features and a linear head trained with focal loss. This places us on the scale between the ResNet-50 blood F1 of 0.14 reported in Galar and VISTA's undecoded 0.3002 [email protected].

The strongest existing method is VISTA. Its reported numbers are temporal [email protected] of 0.3530 and [email protected] of 0.3235 on the hidden NaviCam test set, and validation [email protected] of 0.4730. Its precise validation split may not be reproducible. We will therefore reimplement it with the same backbones, the five-head focal/asymmetric ensemble, validation-guided fusion, and the anatomy grammar plus morphological decoding with tuned windows. This reimplementation runs on our video-level cross-validation folds of the 80 Galar videos. It receives the same hyperparameter search budget, measured in trials and GPU-hours, as our model, with all thresholds and filter widths tuned on inner validation folds. Beating it means a fold-averaged temporal [email protected] gain larger than the across-fold standard deviation. It also means not losing at [email protected], so that the gain does not come from looser boundaries. If the challenge server still accepts submissions, we will additionally report hidden-test numbers against 0.3530.

Three ablations isolate the key ingredient. The first, a post-hoc semi-Markov decoder, uses identical durations and reliability, but the frame head is trained with focal loss alone and the decoder is fit afterward. It tests whether training through the decoder matters. The second sets $r_t \equiv 1$ to test whether reliability gating matters. The third replaces the multi-scale head with the DINOv3 class token to test the spatial component.

Why not just…?

Why not just fit a better post-hoc decoder, such as an HSMM, on top of VISTA-style frame scores? This is the most serious threat. It may capture most of the gain, since VISTA's own ablation shows decoding is where the leverage is. Our first ablation is exactly this model with the same duration family and reliability input. If it matches the end-to-end model within fold noise, we will report the contribution as the reliability-gated duration decoder, which is still a clear advance over fixed morphology, and drop the end-to-end claim.

Why not just train a temporal transformer or temporal convolution over frozen frame features and threshold its output? A sequence model can learn implicit smoothing. However, it is trained with a frame-level loss, so it inherits the swamping of the rare class. It also has no explicit way to require minimum event durations. We include it as a baseline at matched parameter count and with VISTA's decoding on top. If it closes the gap, the semi-Markov structure is unnecessary. The diagnostic we watch is recall on events shorter than the morphological window.

Why not just mask frames flagged as bubbles or dirt and then apply VISTA's filters? This uses the same artifact labels at almost no cost. Hard masking, however, discards frames where a lesion is visible through partial debris, and it does nothing about artifacts the classifier misses. We run it with an artifact classifier trained on identical labels. Reliability gating should win specifically on events that overlap annotated artifact frames. If it does not, soft gating adds nothing.

Why not just feed DINOv3 higher-resolution crops or tiles instead of adding a multi-scale head? Tiling is a strong, simple way to see small lesions, and the file reports no capsule-endoscopy result either way. We compare a 2×2 tiling baseline at equal or higher compute. If tiling matches the attention head, we adopt tiling, because the claim of the paper does not depend on the spatial module.

Proposed Experiments and Decisive Checks

The main study is 5-fold video-level cross-validation on Galar, using temporal mAP at 0.5 and 0.95 IoU as in the RARE-VISION protocol. We report per-pathology event recall at fixed false events per hour, because clinical review cost scales with false events per hour of video.

Three checks decide the proposal. The first is false-positive events per hour inside annotated artifact runs; the gated model should cut these relative to both VISTA-style decoding and hard masking. The second is recall on events of three frames or fewer, where morphological closing is expected to fail and learned durations should not. The third compares end-to-end training against the post-hoc decoder on identical inputs, which tests the central claim directly.

A secondary check concerns the class-imbalance hypothesis. We compare the gradient share received by event frames under segment likelihood and under focal loss. We also report performance as training videos are subsampled, to see whether the segment loss is more label-efficient.

A negative outcome on the third check while the first two succeed would still support a narrower and useful result: learned, artifact-aware duration decoding outperforms morphology.