Compression Surprise Maps: Entropy-Model Mismatch as a Free Abnormality Signal for Chest Radiographs

A learned image codec compresses an image by passing it through a neural encoder, rounding the result to integers (the "latents"), and coding each latent with a probability model called the entropy model. The number of bits spent on a latent is roughly minus the log of the probability the entropy model assigned to it. The compression literature treats the gap between the entropy model's predicted distribution and the true one as pure waste. This note argues that, for medical images, the location of that gap is itself useful. Suppose an entropy model is fitted only to radiographs of healthy anatomy. It will spend unusually many bits exactly where an image departs from healthy anatomy. Every image sent through a telemedicine link already pays for this computation, so the per-location bit cost amounts to an abnormality map that costs almost nothing extra.

The proposal is to build and test "surprise maps." These are per-location log-likelihood ratios between a healthy-anatomy entropy model and a reference entropy model, both scoring the same latents during encoding. The maps need no disease labels and no concept vocabulary, and they need no second network pass. They answer two problems the clinical explainability literature keeps raising: explainers are too expensive to run inside the reading workflow, and they are blind to findings outside a predefined label or concept set. The central empirical question is whether this nearly free signal localizes pathology on chest radiographs, including disease classes held out from any supervision, well enough to compete with supervised attribution methods. It must also clearly beat the obvious unsupervised alternative, which is autoencoder reconstruction error.

Core Claim

Raw bit cost is a poor abnormality signal. It is high wherever an image is complex, such as rib edges, text labels, surgical clips and lines. The claim is that a ratio of two entropy models cancels this generic complexity and leaves the content that is atypical for healthy anatomy. Let $y_i$ be the $i$-th quantized latent from a shared, frozen encoder. Let $p_N$ be an entropy model fitted to latents of "No Finding" radiographs, and $p_R$ a reference entropy model fitted to latents of all radiographs. The surprise at location $i$ is

$$s_i = -\log_2 p_N(y_i \mid c_i) + \log_2 p_R(y_i \mid c_i),$$

where $c_i$ is the conditioning information each model uses. That information is the hyper-latent side information of Joint Autoregressive and Hierarchical Priors plus a parallel spatial context. The $p_N$ term is the bits the healthy model would spend. Summed over the image, $s$ is an image-level log-likelihood ratio that can be used for triage. Mapped spatially, it is a localization cue. Both terms score the same integers, so $s_i$ is a difference of two numbers the encoder can compute in the same pass. At the operating point this adds one extra entropy-parameter head, not a second encoder or any gradient computation.

This differs from the closest existing work in a specific way. NICE learns classifier-driven explanation masks and uses them to downsample background pixels. It has no probabilistic entropy model, and its masks do not transfer to classes unseen in training. DMIC uses a coronary segmentation model to reward bit allocation inside HEVC, and it produces no explanation for the clinician. Here the direction runs the other way: the compression model's own probabilities are the explanation, and no downstream task model is involved.

Research Program

The build starts from a mean-and-scale hyperprior codec, following Minnen et al., trained on NIH ChestX-ray14 (Wang et al.). Parallel context is used instead of serial autoregression, following the fast-context idea of Li, Ma et al., so encoding time stays compatible with the workflow. The encoder, decoder and hyper-encoder are then frozen. Two entropy heads are fine-tuned: $p_N$ on patients with no finding, and $p_R$ on all training patients. The split is by patient.

Three design issues from the compression literature directly affect whether $s_i$ is meaningful.

The first is the gap between training and testing. Training simulates rounding with uniform noise, while deployment uses hard rounding. Zhang et al. and Zafari et al. show this gap distorts rate estimates at low bitrate. A distorted estimate would add spurious surprise in flat lung fields, so both heads receive a final fine-tuning stage with straight-through hard rounding.

The second is the shape of the distribution. A Gaussian head facing a heavy-tailed latent will "surprise" on every large coefficient. The Gaussian, the three-component mixture of Cheng et al., and the element-wise generalized Gaussian of Zhang et al. are therefore compared as the parametric family inside both heads. The hypothesis is that more flexible families reduce false surprise from ordinary texture.

The third is that the hyperprior can absorb abnormality. The side information $z$ is computed from the image itself, so a lesion can be "explained away" by the hyper-latents. Two variants are tested. In the first, both heads share one $z$. In the second, the surprise is computed with the healthy model's own hyper-decoder, and the bits of $z$ itself are added into the map at their spatial footprint.

A secondary use follows directly from the direction on distribution mismatch. When the total $\sum_i s_i$ is large, the lossy codec is operating off the distribution it was tuned for. That is exactly when rate-distortion guarantees are weakest. The image-level score is evaluated as a trigger for falling back to lossless transmission. The evaluation checks whether images the trigger flags suffer disproportionate loss in downstream classifier accuracy after lossy coding.

Baselines

The naive reference point is the decoupled pipeline. The codec transmits the image and no map is produced. At the workstation, a separately trained classifier with post-hoc Grad-CAM supplies localization. For Grad-CAM, Khakzar et al. report a mean IoU of 0.077 on ChestX-ray8. The second naive point is the raw bit map $-\log_2 p_R(y_i)$ alone. This ablation isolates the key ingredient, the ratio. If raw bits localize as well as $s_i$, the healthy-anatomy model adds nothing.

The strongest supervised competitor is Multi-layer Information Bottleneck Attribution from Khakzar et al.. It inserts noise into intermediate classifier layers to measure which regions carry label information, and they report a mean IoU of 0.189 on ChestX-ray8. It is the strongest because it is the best attribution number in the file on this exact benchmark. The paper also shows that supervised signal localizes far better than gradient saliency. Region Verification from Zhang et al. is a second supervised reference. It crops salient regions and re-classifies them to reject artifacts, and it reports a mean AUC of 0.855, which is a classification number, not localization. Both supervised methods are run on the same 880 bounding-box images, with DenseNet-121 classifiers trained on the same patient split and given the same hyperparameter search budget as the codec heads. Maps are thresholded by the same rule, a sweep of percentile thresholds, reporting the best and the validation-selected value for every method.

The strongest unsupervised competitor is reconstruction error. The absolute difference between the input and the decoded output of the same codec is computed, and also that of a codec trained only on healthy images. It uses identical data and compute and requires no labels, so it is the fair head-to-head for the central claim.

Beating the baselines means the following. $s_i$ must clearly exceed raw bits and reconstruction error on IoU and pointing accuracy. It must exceed Grad-CAM's 0.077. On disease classes held out entirely from classifier training, it must match or exceed Multi-layer IBA, because there IBA must rely on a classifier that never saw the class. On seen classes, falling short of 0.189 is expected, and the result is reported as a cost-versus-accuracy trade-off rather than claimed as a win.

Why not just…?

Why not just use reconstruction error from an autoencoder trained on healthy images? This is the most serious threat, since it is the textbook unsupervised anomaly map and costs the same. The argument against it is mechanistic. A codec reconstructs whatever it is given, lesions included, at the price of more bits. Reconstruction error therefore tends to track what the decoder cannot express, such as fine texture and edges, rather than what is improbable under healthy anatomy. Nothing in the file measures this on radiographs, so the head-to-head in the baselines section settles it. If reconstruction error matches $s_i$, the proposal reduces to "anomaly detection with a codec," and the contribution shrinks to the cost argument.

Why not just run a supervised classifier with the best attribution method? On the labeled classes that will likely localize better, since Khakzar et al. reach 0.189 IoU. It fails in two places. It is bound to its label set, which mirrors the concept-bank incompleteness that Mehta et al. acknowledge for concept bottlenecks. It also needs a second model and extra passes at the workstation, a cost Krebs et al. quantify for perturbation methods: about 27 minutes and 36 GB per slide for occlusion, versus about 2 minutes for HiResCAM. The held-out-class experiment and the wall-clock measurement test exactly these two points.

Why not just steer bits with a segmentation or saliency model, as DMIC or NICE do? DMIC reports a 47.594% BD-rate saving over HEVC on ImageCAS. That result answers a different question, efficiency for one known target, and its authors note it is tuned to a single diagnostic task. NICE shows class-dependent masks that fail on unseen classes, and Zhang et al. show that saliency-driven cropping degrades on diffuse pathology. Neither gives the reader an explanation. Surprise maps are complementary and could later supply the allocation signal, but that is out of scope here.

Proposed Experiments

The primary experiment is localization on the 880 radiologist bounding boxes of ChestX-ray8. Four quantities are reported: IoU, pointing accuracy, and IoBB accuracy at T=0.1 for comparison with Wang et al., who report 0.99 for cardiomegaly with a supervised model. These are computed for all baselines and for every combination of entropy family and hyperprior variant.

The decisive held-out test removes two classes from all supervised training, for instance pneumothorax and nodule. It then compares $s_i$ with classifier-based attribution on those classes only. The prediction is that the surprise map's advantage over IBA grows sharply there. If it does not, the label-free argument fails.

Image-level triage is measured by AUROC of $\sum_i s_i$ for abnormal versus no-finding images. Two related checks use the same score. On BrixIA, the rank correlation between $\sum_i s_i$ and the cumulative severity score tests whether surprise scales with disease extent. The lossless-fallback test measures the drop in classifier AUC after lossy coding, split by the trigger's decision.

Compression is also reported: BD-rate of the codec on ChestX-ray14 against VVC and against the same codec trained on Flicker2W. This quantifies the natural-to-medical mismatch in bits. No reported result for learned codecs on chest radiographs was found on Lacuna. Added latency and memory for computing $s_i$ are measured against Grad-CAM and Multi-layer IBA on identical hardware.

Finally, a small reader pilot is run with the explicit caveat that it is not a prospective trial. Radiologists read a held-out set with and without surprise overlays, and time-to-decision and missed findings are recorded. It is aimed squarely at the gap Song et al. describe, where faithful-looking maps do not improve decisions.

The project is falsified if either of two results appears. The first is that the ratio fails to beat raw bits and reconstruction error on localization. The second is that the held-out-class test shows no advantage over supervised attribution.