Does Pixel Reconstruction Teach Small Medical Encoders to Memorize the Scanner?

When a model is pre-trained on a few hundred echocardiograms by hiding parts of each image and asking it to fill them in, the cheapest way to fill them in is to learn what that particular ultrasound machine's speckle, gain and texture look like. In a small cohort from one or two sites, those low-level acquisition details are exactly the features a network later uses as shortcuts, and they are the first thing to break when the scan comes from a different device. This proposal tests one specific remedy. It keeps masked pre-training but changes what the model predicts for the hidden regions: instead of raw pixels, a separate small predictor reconstructs the encoder's own feature representation of those regions. The prediction is attached to the backbone's output feature map, so the same objective can be used with a CNN, a U-Net or a Vision Transformer.

The question is narrow and falsifiable. Under small-cohort pre-training, does moving the reconstruction target from pixels into feature space reduce how much acquisition-site information the encoder stores? If it does, does that reduction show up as a smaller drop in segmentation accuracy across devices, and does it hold across backbones? If site information is just as decodable either way, the hypothesis is wrong. That outcome would also be useful, because the medical literature currently treats "decoupled" pre-training as helpful without saying why.

Core Claim

Decoupled masked modeling is the line of work that separates the encoder, which should learn semantics, from the module that recovers masked content. In vision, Context Autoencoder has its encoder see only visible patches. A cross-attention regressor predicts the latent features of masked patches, and a separate decoder handles pixels. CAE reports 50.2 mIoU on ADE20K against 48.1 for a standard masked autoencoder. In text, CrossBERT offloads masked-token recovery to a shallow predictor and more than doubles retrieval quality relative to BERT. Both papers motivate the proposal: they argue that coupling representation to low-level reconstruction wastes encoder capacity on local detail. Both were tested at large scale on natural data, where "local detail" is benign.

Small medical cohorts are where that wasted capacity becomes harmful. Three papers establish the risk this proposal targets. Unveiling COVID-19 from Chest X-ray showed small-data models clustering images by hospital and hardware rather than pathology. Shortcut Learning in Medical Image Segmentation showed segmentation networks exploiting calipers, text and padding. Foundation versus Domain-Specific Models showed a domain-specific echo model falling from 0.893 to 0.667 Dice on noisy point-of-care ultrasound while a broadly pre-trained model held 0.857.

Our claim is that pixel-target pre-training on a small cohort actively pushes acquisition texture into the encoder. A latent-target objective, whose targets come from a slowly updated copy of the encoder, removes that pressure. The objective we test, for an image with masked region set $M$, is

$$ \mathcal{L} = \frac{1}{|M|}\sum_{i \in M} \left\| g\big(f_\theta(x_{\setminus M})\big)_i - \mathrm{sg}\big[f_{\bar\theta}(x)\big]_i \right\|^2 . $$

Here $f_\theta$ is any backbone producing a spatial feature map, $x_{\setminus M}$ is the image with the masked regions removed or zeroed, $g$ is a shallow cross-attention predictor, $f_{\bar\theta}$ is an exponential-moving-average copy of the encoder, and $\mathrm{sg}$ means no gradient flows into the target. No pixel loss is used in the main arm.

The closest existing work is EDMAE. EDMAE also aligns latent features between a teacher branch and a student branch on echocardiography, but it differs from this proposal in three ways. It keeps a pixel reconstruction loss. It is fixed to a DenseNet encoder. And it pre-trains on a large private pediatric dataset without ever measuring shortcut or site information. We isolate the pixel loss as the variable of interest, vary the pre-training cohort size downward, and measure site encoding directly.

Research Program

The work has three parts.

Shared training harness. We build one pre-training harness in which masking, the predictor and the target are identical across backbones and only the target type varies. Every backbone produces a feature grid, and the predictor reads that grid. This is the interface used by SmartPretrain, which applied a model-agnostic self-supervised objective to output embeddings of Transformers, GNNs and CNNs for motion prediction. We borrow its interface idea, not its domain. We run three backbones:

  • ResNet-18 U-Net, motivated by the finding in Unveiling COVID-19 that ResNet-18 beat ResNet-50 on small data;
  • DenseNet, as in EDMAE;
  • ViT-Small.

For CNNs, masked regions are zeroed at the input. The research file notes that this approximation can reintroduce local spatial bias, so we report CNN and ViT results separately rather than averaging them.

Measuring stored site information. We measure how much acquisition-site information each pre-trained encoder stores using a frozen linear probe. The probe is trained to predict source dataset or device from pooled features. Its accuracy is reported alongside a probe for the clinical target, such as view or chamber, so that "less site information" cannot simply mean "less information."

Downstream evaluation. We relate both probe results to downstream left-ventricle segmentation under small labeled fractions and under cross-site transfer.

Baselines

Training from scratch. The naive reference is the same segmentation network trained from random initialization, with the shortcut hygiene described below. DenseMP shows how badly this can do in few-shot medical segmentation: 31.86% Dice on abdominal CT without dense pre-training versus 72.84% with it. It anchors the scale of any pre-training gain.

Contrastive pre-training (SimCLR). The strongest existing competitor is SimCLR pre-training followed by fine-tuning, as in Is Contrastive Learning Suitable for Left Ventricular Segmentation. That paper reports 0.9125 Dice on EchoNet-Dynamic with only 5% labels, 0.9252 with full labels, and 0.9311 Dice on CAMUS. It found that the non-contrastive BYOL failed on ultrasound speckle, which is directly relevant because our latent target also uses a moving-average teacher and could collapse similarly. We rerun SimCLR ourselves on the same pre-training images, with the same backbones, epochs and tuning budget, rather than quoting the reported numbers, because our cohorts are smaller. The quoted 0.9125 serves only as a sanity check that our reimplementation is competitive at 5% labels on EchoNet-Dynamic.

Coupled masked autoencoder. The standard pixel-reconstruction masked autoencoder, trained on the same data, with the same backbones and the same tuning budget.

EDMAE. Reimplemented, including both its latent alignment loss and its pixel loss. EDMAE reports 93.09% Dice on CAMUS after pre-training on a large private set.

Fine-tuned SAM. We report the published fine-tuned SAM figure as an external reference, not as a matched run, because it uses expert box prompts at test time. Foundation versus Domain-Specific Models reports 0.891 Dice on the CAMUS two-chamber view.

Key ablation. The ablation that isolates the proposal's ingredient holds encoder, masks, predictor depth and compute fixed and varies only the target: pixels, latent, or latent plus pixels.

What counts as success. Beating the baselines means two things at once:

  • a statistically lower site-probe accuracy and a smaller cross-site Dice drop than both the pixel-target arm and SimCLR;
  • in-domain Dice at 5% labels no worse than SimCLR's within seed variance.

Why not just…?

Why not just clean the images of known shortcuts? For visible artifacts, cleaning works. Shortcut Learning in Medical Image Segmentation reports that removing calipers and using random crops lifts affected segmentations from roughly 70–80% to 90–98% Dice. We therefore apply that hygiene to every arm, so it is not a confound. Cleaning cannot remove vendor speckle statistics, gain curves or probe-specific texture, because these cover the whole image and are not separable from anatomy. The controlled experiment below injects exactly such a shortcut. If cleaning alone already closes the cross-site gap on real data, the proposal's practical value shrinks to the synthetic setting, and we will report that.

Why not just use SimCLR with strong augmentations? This is the real threat. Contrastive learning discards whatever its augmentations vary. If we augment gain and speckle, SimCLR may become invariant to the very site texture we target. The difference is that contrastive invariance requires knowing the nuisance in advance, whereas the latent-target objective is meant to avoid encoding nuisances that are hard to predict from context. We run SimCLR both with standard augmentations and with ultrasound-specific intensity and speckle augmentations. If the augmented SimCLR matches our method on site probing and cross-site Dice, the honest conclusion is that augmentation design, not decoupling, is what matters.

Why not just give a coupled masked autoencoder a deeper decoder? A standard masked autoencoder already has a separate decoder. One could argue that its depth, not the target type, is what keeps detail out of the encoder. The target ablation addresses this directly. We run the pixel-target arm with a predictor and decoder whose depth and parameter count match the latent arm's. If depth explains the effect, the pixel arm with a deep decoder will match the latent arm.

Why not just fine-tune a large foundation model? SAM transfers well under device shift, but the reported results depend on clinician-drawn boxes at test time and on pre-training far beyond the scale of any clinical group. Our question is what to do when only a few hundred local scans exist. We include the published SAM figure as a ceiling reference rather than a competitor.

Proposed Experiments

Pre-training regime. We pre-train on CAMUS images (500 patients, two views) at cohort sizes of 50, 100 and the full 500 patients. Separately, we pre-train on comparable-size subsets of EchoNet-Dynamic. The two datasets come from different institutions and equipment, which gives a natural site shift without private data.

Downstream evaluation. Fine-tuning for left-ventricle segmentation uses 1%, 5% and 10% of labels and is evaluated three ways: in-domain, cross-dataset (pre-train and fine-tune on one dataset, test on the other), and on the CAMUS two-chamber view held out as an out-of-distribution view. We report Dice and Hausdorff distance, with five seeds per configuration.

Site-probe test. For each frozen encoder, we report the frozen-feature site-probe accuracy and view-probe accuracy, then correlate site-probe accuracy with cross-dataset Dice drop across all arms and backbones. A clear negative relationship, with the latent arm sitting lowest on site information, supports the mechanism. No difference between pixel and latent arms falsifies it.

Controlled shortcut test. This is the decisive check. During pre-training and fine-tuning, half of one cohort receives a faint synthetic speckle-filter signature, and that signature is correlated with a downstream label we construct, such as end-diastole versus end-systole frame. At test time the correlation is reversed. We measure how much accuracy each arm loses when the shortcut flips. This isolates whether the pixel target specifically drives the memorization of texture-level shortcuts.

Cross-backbone check. Finally, we test whether the ranking of target types is consistent across ResNet-18, DenseNet and ViT-Small. A consistent ranking supports the claim that the latent target is an architecture-agnostic fix rather than a property of one backbone.