Compress Until the Experts Would Disagree

When medical images are streamed live from where they are captured to where they are read, the link is slow and every frame must be sent without waiting to batch it. Examples are a handheld ultrasound probe talking to a hospital server, or a slide scanner feeding a remote pathologist. Today's low-delay compressors measure success against a single correct answer: either pixel error, or whether the decoded image still produces the same diagnostic label. For many medical findings no single correct answer exists. Pathologists grading the same prostate biopsy, or sonographers judging the same lung clip, routinely disagree, and that disagreement is itself clinical information about borderline disease. This proposal asks the compressor to preserve the set of diagnoses experts would plausibly give for each frame, and not a majority-vote label. It spends bits only where compression would change that set.

Concretely, we take online conformal compression, a zero-delay method that adapts its bit rate on the fly so that the long-run fraction of "outages" (frames decoded wrongly) provably stays below a target $\alpha$. We redefine an outage as a change in the plausible-diagnosis set computed by a virtual panel of rater models. Because this outage is still a yes/no event over a discrete set, the existing per-sequence guarantee carries over unchanged. We therefore avoid the open problem of extending conformal compression to continuous distortions. Our prediction is that consensus-based outage definitions waste bits on exactly the frames where experts split. On those frames the majority label flips under tiny perturbations, so it is an unstable target. Our definition stops paying for that instability.

Core Claim

Let a panel of $K$ rater models produce a label distribution $p(y \mid x)$ for frame or tile $x$, and define the plausible set $S(x) = \{y : p(y \mid x) \ge \tau\}$, the labels that at least a fraction $\tau$ of the panel would assign. After zero-delay encoding and a possibly lossy channel, the receiver holds a reconstruction $\hat{x}_t$. We declare an outage at time $t$ when

$$ e_t = \mathbf{1}\left[\, S(\hat{x}_t) \neq S(x_t) \,\right], $$

so an outage is flagged both when compression erases genuine ambiguity and when it invents ambiguity that was not there. The encoder picks the rate for each frame from a ladder of codec operating points using a calibration variable $\gamma_t$. It updates this variable online as $\gamma_{t+1} = \gamma_t + \eta\,(e_t - \alpha)$, which is the error-tracking update used in online conformal compression. The resulting guarantee is that the long-run outage rate is at most $\alpha$ plus a term shrinking as $1/T$, for any sequence and without distributional assumptions. This bound comes from the standard online conformal argument, since $\gamma_t$ is bounded.

The claim is not only that this definition works. It is that, at a matched guarantee, it uses measurably fewer bits than a consensus-label outage, and that the savings concentrate on high-disagreement content.

Relation to Prior Work

The mechanism comes directly from Ganesan et al.'s Online Conformal Compression for Zero-Delay Communication with Distortion Guarantees and Prediction-Powered Communication with Distortion Guarantees. Those papers pair an autoregressive predictor with online calibration and handle Gilbert-Elliott channels, which alternate between good and bad states, through a virtual error queue. They were evaluated only on text tokens with one correct token per step, and they name continuous or clinical distortions as open.

We keep their guarantee and update rule but change what counts as an error. The outage is anchored to a set-valued target derived from multiple raters. This is also why we stay within 0–1 loss rather than attacking the continuous extension.

The rater panel comes from the multi-rater literature, which motivates the whole proposal by showing that disagreement is signal:

None of these considers transmission. We use them only as the receiver-side and encoder-side judge of "diagnostic sameness."

The closest task-aware idea is Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle. It fuses rater masks to maximize a downstream classifier's AUC, but its fused target is tied to one classifier and assumes static, centralized data. We do not fuse raters into a single target. We keep the set.

Classical real-time theory constrains the design. Distortion-transmission trade-off in real-time transmission of Markov sources shows that for Markov sources the optimal sender transmits only when estimation error crosses a threshold, a rule we test as a baseline. Optimal design of sequential real-time communication systems shows that exact optimal design over high-dimensional belief states is intractable, which is why we accept an online calibrated rule rather than a dynamic program.

Research Program

The system has three parts.

The first is an encoder-side codec with a small ladder of rates. We use a learned image codec at around six quality levels, plus a "skip and hold previous frame" option, and add a lightweight copy of the rater panel. On a slide scanner this panel can be full size. For handheld ultrasound we distill it. Because the reconstruction is a deterministic function of received bits, the encoder can replay the decoder given acknowledgment feedback. It can therefore compute $e_t$ itself, as online conformal compression already assumes.

The second is a receiver running the full virtual panel: per-rater models in the style of Grove et al. for classification, and Pionono for segmentation, where $S$ is defined per region.

The third is the calibration loop above, run over simulated Gilbert-Elliott erasure channels as in the online conformal compression papers.

We will analyze three questions. First, where the bits go, broken down by panel entropy on the original content. Second, how the choice of $\tau$ trades bitrate against faithfulness to rare minority opinions. Third, whether the plausible-set outage tracks what human readers see.

For the third question, a small reader study will show original and reconstructed tiles, both selected near the rate chosen by our method, to pathologists. Readers will not know which is which. We measure whether their label sets change. This check matters because the guarantee is stated with respect to a machine panel, which is only a proxy for humans.

Baselines

The naive reference is a fixed-quality codec. Every frame is sent at one operating point, chosen offline on a development split so that its average consensus-label outage matches $\alpha$. It shows what adaptivity buys. It runs on the same codec ladder and data, and no reported result is found on Lacuna for fixed-rate codecs on these medical datasets.

The strongest existing competitor is online conformal compression with its original single-truth outage. Here the target is the panel's majority label, and an outage is $\mathbf{1}[\arg\max p(\cdot \mid \hat{x}_t) \neq \arg\max p(\cdot \mid x_t)]$. This baseline has the same machinery and differs from our method only in the error definition, so the comparison is exact. It uses the same codec ladder, panel, channel traces, step size $\eta$ swept over the same grid, and target $\alpha$. On text, the source paper reports a 30% bitrate reduction in synchronous mode and 16% in asynchronous mode against Dropout-LLMZip at $\alpha = 0.2$. No reported result is found on Lacuna for it on medical images.

Beating it means two things at the same achieved $\alpha$:

  • lower bits per tile, with confidence intervals over sequence orderings;
  • equal or lower change in the panel distribution between original and reconstruction, measured by KL divergence and calibration error against held-out raters.

The event-triggered baseline follows Chakravorty and Mahajan. The sender transmits at high rate only when the pixel-level error of the held frame exceeds a threshold, with the threshold tuned for the same average bit budget. It represents the classical signal-fidelity answer.

The key ablation removes online calibration and fixes the rate rule from a development split while keeping the plausible-set definition. It isolates whether the guarantee survives distribution shift or whether the definition alone does the work.

Why not just…?

Why not just keep a single majority-vote label and use online conformal compression as published? This is the main threat. If the majority label rarely flips under compression, the two definitions coincide and our method saves nothing. The consensus baseline above settles it directly. Our method should save bits, and the saving should concentrate on tiles with high panel entropy. If the bitrate gap is flat across entropy levels, the idea fails.

The Grove et al. result says a consensus target misrepresents borderline cases: calibration error rose from 0.050 to 0.183. That is evidence the majority target is fragile, not proof it is costly to transmit.

Why not just scale the pixel-error threshold by how much raters disagree? This heuristic, "tolerate more error where experts split," is cheap and plausible. We include it as a variant of the event-triggered baseline, with a threshold proportional to panel entropy. It lacks a guarantee, however, and pixel error is a poor proxy for label change on tiny structures. FUSEP: A Multi-Center Benchmark for Diverse Tasks in Early Pregnancy Fetal Ultrasound Screening notes that key markers occupy 0.2–0.5% of the frame. If this heuristic matches our bitrate and its outage rate stays under $\alpha$ across shifts, we would report that the simpler rule suffices.

Why not just run the diagnosis at the edge and send only the label set? If only the machine verdict mattered, that would be optimal. But AI Enhances Lung Ultrasound Interpretation Across Clinicians with Varying Expertise Levels shows the value of AI lies in helping humans read images, raising reader Fleiss' κ from 0.427 to 0.756 on consolidation. Remote review therefore needs the image itself.

Our stream serves that reviewer, and the panel acts as a judge of whether the reviewer's plausible readings are preserved. The reader study tests whether this proxy is faithful. If humans' label sets change where the panel's do not, the premise weakens.

Proposed Experiments

The primary testbeds are multi-rater pathology sets streamed tile by tile in scan order, mimicking remote slide review:

  • Gleason2019, with 6 pathologists and per-region labels, using Pionono per region;
  • ISUP grading, with 20 pathologists at slide level, using a per-rater panel;
  • DEPP, with about 11 pathologists across four classes, where we explicitly keep the "indefinite for dysplasia" cases that Grove et al. had to exclude, since set-valued targets handle them naturally.

As a sequential imaging check we stream QUBIQ brain MRI slices with 7 experts. Distribution shift is induced by switching the stream between staining sources or centers partway through, mirroring the Shakespeare-to-lyrics shift in the source paper. Channels are Gilbert-Elliott traces at several erasure rates, in both synchronous and asynchronous modes.

We report four families of metrics:

  • bits per tile against achieved outage rate, including the running trajectory over time;
  • the bitrate breakdown by panel entropy;
  • KL divergence between panel distributions on original and reconstructed content, and ECE against held-out raters;
  • downstream κ of the receiver panel.

The decisive checks are three:

  • The guarantee holds after the shift for the calibrated method but not for the fixed-rule ablation.
  • At a matched $\alpha$, the plausible-set method uses fewer bits than the consensus-outage baseline, and the gap grows with panel entropy.
  • In the reader study, pathologists' label sets on reconstructions sent at our chosen rates change no more often than $\alpha$.

Ultrasound deployment with a distilled encoder-side panel on FUSEP-style vendor shift is a follow-up. FUSEP's labels are consolidated, so it cannot test the multi-rater claim.