Temporal Disagreement as a Label-Free Signal for Calibrating Low-Bit Video Segmenters

Shrinking a neural network to 2–4-bit integers is how a colonoscopy polyp segmenter would fit on the small processor inside an endoscopy tower. Quantization is the step that rounds the network's weights and intermediate activations to a few integer levels. The methods that tune this rounding were built and tested on image classification, where only the order of the class scores matters. They are known to break on dense, per-pixel tasks: small targets vanish and boundaries drift. Fixing that normally requires labeled target data, and in clinical video that means expert-traced masks. Those are the scarcest resource in the field.

We propose using the video itself to supply most of the missing labels. Consecutive endoscopy frames mostly show the same tissue. When two nearly identical frames receive different masks from the quantized model, most of that disagreement comes from rounding noise, not from a change in the scene. We claim that disagreement on carefully chosen near-duplicate frame pairs measures one specific part of quantization error: the part that changes from frame to frame. A handful of expert-labeled anchor frames per hospital can then correct the remaining part, the systematic bias that consistency cannot see. Combining the two lets a low-bit segmenter tune its quantizer settings for a new site with almost no new annotation. The tuning uses forward passes only, so it is cheap enough to run on the device.

Core Claim

Write the quantized model's per-pixel prediction on frame $t$ as $\hat y^q_t = \hat y_t + b_t + r_t$. Here $\hat y_t$ is the full-precision prediction, $b_t$ is a slowly varying systematic error (from clipping and scale mismatch, shared across neighboring frames), and $r_t$ is a rapidly varying error. The rapid part arises because small changes in pixels or motion push activations across rounding boundaries. For a frame pair whose true content barely changes, the expected squared mask disagreement is approximately

$$ \mathbb{E}\,\|\hat y^q_t - \hat y^q_{t+1}\|^2 \;\approx\; \|\hat y_t - \hat y_{t+1}\|^2 + \mathbb{E}\,\|r_t - r_{t+1}\|^2 . $$

The first term is small by construction, and the systematic terms cancel. So disagreement observes the frame-varying error $r$ and is blind to $b$.

This decomposition gives a precise division of labor between unlabeled video and expert time. Unlabeled redundancy calibrates away flicker. A few anchors fix bias. It also predicts a non-obvious selection rule. Perfectly static pairs carry no signal, because deterministic rounding of identical inputs gives identical errors. Pairs with too much change violate the shared-content assumption. The informative pairs therefore lie in a middle band of inter-frame change, and choosing them is the job of a gate that monitors frame-to-frame continuity.

Prior work motivates this claim without making it. PMQ-VE shows that inter-frame feature variation defeats a single global clipping bound in low-bit video enhancement models. It answers with frame-specific bounds and distillation from a full-precision teacher and an 8-bit teacher. It uses full supervision, evaluates only reconstruction quality (PSNR/SSIM), and never uses cross-frame agreement as an objective. RDO-Q allocates per-channel bit-widths by assuming total distortion is the sum of independent channel distortions. Our decomposition separates exactly the correlated-across-time component that such an additive, per-frame view ignores.

Research Program

The work has three stages. The first stage is purely diagnostic and can kill the project.

Stage 1: test the decomposition. On SUN-SEG, which has dense masks for every frame, we take a full-precision video polyp segmenter and quantize it with standard post-training quantization at W8A8, W4A4, W3A3, and W2A4. We then measure the error against ground truth directly and split it into a component shared across a sliding window (an estimate of $b$) and a residual (an estimate of $r$).

We then ask whether the label-free disagreement on gated frame pairs ranks quantizer configurations the same way true residual error does. We ask the same question of the error on small polyps, using SUN-SEG's small-object and fast-motion attributes. The kill criterion is a rank correlation below about 0.5 between disagreement and true error at 4 bits and below. In that case consistency is not a usable calibration signal, and the project reduces to the anchor-only baseline. We also measure how much of the total error $r$ accounts for. If bias dominates, consistency can help only marginally, and we will report that honestly.

Stage 2: calibration objective and optimizer. We tune only the quantizer's parameters $\theta$, meaning per-layer clipping bounds and per-channel scales, and leave the weights frozen. We minimize

$$ L(\theta) = \sum_{(t,t+1)\in S} d\big(q_\theta(x_t), q_\theta(x_{t+1})\big) + \lambda \sum_{a \in A} \ell\big(q_\theta(x_a), y_a\big). $$

The symbols are defined as follows. $q_\theta$ is the quantized segmenter. $S$ is the set of frame pairs accepted by the continuity gate. $d$ is a soft-mask disagreement weighted toward predicted foreground, so the empty background does not dominate. $A$ is a small set of expert-labeled anchor frames from the target site, with masks $y_a$. $\ell$ is Dice loss, and $\lambda$ balances the two terms.

The gate reuses the idea behind VPSentry. VPSentry monitors inter-frame continuity to stop camera jerks and blur from polluting its memory of past frames. We repurpose that monitor to select pairs in the informative middle band of change.

Because rounding is a staircase function with zero gradient almost everywhere, we optimize with the forward-only two-pass gradient estimate used by ZOA. ZOA estimates a gradient by perturbing all parameters at once and comparing the loss across two forward passes. It adapted quantized classifiers down to W3A3 in about 103 MB of memory. Restricting the search to quantizer parameters (thousands of values rather than millions of weights) is deliberate. The noise of this kind of gradient estimate grows with the number of parameters, so it is the regime where forward-only optimization of a dense model is plausible.

Stage 3: deployment protocol. Calibration for a new hospital uses unlabeled clips from that hospital plus a fixed anchor budget: one expert-masked frame per clip, or a total of 20 to 50 frames per center. Calibration may then continue online with the consistency term alone. We will check whether online continuation drifts, given the confirmation-bias failures documented for self-training by Can et al. and Endo-SemiS.

The base models will be PNS+ from the SUN-SEG paper and VPSentry, both trained semi-supervised with 10% labels. This keeps the whole pipeline inside the annotation-scarce regime.

Baselines

The naive reference is min-max post-training quantization of the full-precision semi-supervised model, calibrated on a few hundred unlabeled source frames with no target-site tuning. This is what a deployment engineer would do by default, and it sets the floor. The upper reference is the full-precision model itself. For VPSentry trained with 10% labels, the strongest reported number is a Dice of 83.95 on SUN-SEG-Hard, at 28–30 FPS on an RTX 4090 (VPSentry). PSDNet reaches 81.11 in the same setting. For PNS+, the reported number is a Dice of 0.756 on SUN-SEG-Easy Unseen (PNS+). We found no reported result on Lacuna for any quantized video polyp segmenter, so the quantized numbers will be new and must be generated under one shared protocol.

The strongest label-free competitor is teacher-matching calibration. Here the same quantizer parameters are tuned, using the same forward-only optimizer and the same number of forward passes, to reproduce the full-precision model's outputs on the same unlabeled target clips. This is the distillation logic of PMQ-VE and the output-distortion logic of RDO-Q, applied to segmentation. It is the real threat because it also needs no labels. We additionally run PMQ-VE's frame-specific bound initialization as a structural competitor.

The strongest adaptation competitor is ZOA, with its per-pixel entropy objective, run on the same quantizer parameters with the same budget.

The ceiling for quantized models is quantization-aware training with all available SUN-SEG masks.

Three ablations isolate the key ingredients: consistency alone without anchors, anchors alone without consistency, and consistency with the continuity gate replaced by all adjacent pairs.

Beating the competition means the following. At W4A4 and W3A3, the method must recover significantly more of the gap between naive quantization and full precision than teacher matching does, by at least 2 Dice points, with 95% confidence intervals computed over clips. This must hold on SUN-SEG-Hard Unseen and on PolypGen sequence clips. It must also beat anchors-alone at an equal anchor budget, since otherwise the unlabeled video contributes nothing.

Why not just…?

Why not just calibrate against the full-precision teacher on unlabeled target video? This is the alternative most likely to match us. In distribution, it may well do so. Our case rests on two points. First, matching the teacher's outputs or features spreads the effort over every pixel, and in endoscopy those pixels are overwhelmingly background. The small-target collapse seen in quantized dense models by Huang et al. and Park et al. can therefore survive a low reconstruction error. Second, at a new hospital the teacher is itself wrong. TransNetR and the EndoCV double encoder-decoder study document large cross-center drops, so matching the teacher faithfully copies those errors, while anchors can correct them. The PolypGen cross-center experiment settles this. If teacher matching ties us there too, the honest conclusion is that consistency is redundant with distillation, and we will report it that way.

Why not just smooth the masks over time after inference? Post-hoc temporal correction, as in the spatiotemporal module of Endo-SemiS, removes flicker without touching the quantizer. By our decomposition, smoothing averages out $r$. It cannot restore a polyp that the bias term $b$ erases in every frame, and it adds latency. We run a sliding-window smoothing baseline on the naive quantized model. If it matches our Dice on small polyps, then calibration buys nothing beyond flicker reduction.

Why not just apply ZOA as published, with entropy minimization? On segmentation, minimizing entropy rewards confident predictions. When foreground is rare, the cheapest confident answer is "background everywhere," which is exactly the small-polyp failure we are trying to fix. ZOA was evaluated only on classification (ZOA), so this is a hypothesis, not a known result. The same-budget comparison, tracking foreground area over adaptation steps, will show whether entropy collapses or holds.

Why not just deploy at 8 bits? If W8A8 loses less than half a Dice point, the problem exists only at four bits and below. Stage 1 reports the full bit-width curve first, and the claim is scoped to whatever range shows a real gap.

Proposed Experiments

The decisive check is Stage 1: does gated disagreement track true quantization error, including on small polyps, at four bits and below? The main experiment then calibrates at W4A4, W3A3, and W2A4 and compares against every baseline above. It uses three settings: SUN-SEG Easy and Hard (Seen and Unseen); PolypGen sequence clips as an unseen-center target, with anchors drawn from each center; and KvasirCapsule-SEG as a second video modality.

The metrics are Dice and mean IoU overall, Dice on SUN-SEG's small-object and fast-motion attributes, and a flicker score defined as mask change on gated near-duplicate pairs. We also record peak memory and frames per second for the calibration loop.

An anchor-budget sweep from 0 to 50 frames per center traces how many expert labels unlabeled redundancy is worth. A pair-selection sweep across bands of inter-frame change tests the predicted inverted-U: little signal from static pairs, harmful signal from pairs separated by scene changes.

A negative outcome would be informative on either axis. If disagreement fails to track error, or teacher matching ties us under center shift, the proposal fails on its own terms.