Spending the Bus on Detail: Error-Guided Full-Resolution Fetching for Whole-Volume Medical Pretraining

Self-supervised pretraining is what makes medical imaging models usable with few expert labels. A model first learns from large piles of unlabeled scans, typically by hiding parts of an image and learning to fill them in, and then needs only a small labeled set to learn the diagnostic task. The details that matter clinically, such as a small nodule or a thin pneumothorax rim, live at full resolution. A full-resolution CT volume does not fit through a single GPU's memory together with a large model. Practitioners therefore either shrink the whole scan or train on small random crops. Shrinking throws away detail, and cropping throws away anatomical context. Systems researchers working on language models and 3D rendering have shown a third route. Bulky data stays in cheap host memory, a small proxy stays on the GPU, and the proxy decides which few pieces must cross the slow CPU-to-GPU link (the PCIe bus, roughly 32–64 GB/s) in time for the next computation.

We propose bringing that route into medical pretraining with one change: the pieces fetched are chosen by learning value, not by visibility or attention. A coarse copy of every scan stays on the GPU. Each training step, a cheap coarse pass over the whole scan finds where the model currently reconstructs worst. Full-resolution patches for only those regions are pulled from host memory or SSD while the coarse pass is still running. The model then trains its fine-detail reconstruction on exactly those regions. The question is whether this beats downsampling and random cropping in downstream accuracy at 1–10% labels, at equal GPU-hours on one commodity GPU. A second question is whether the fetch can be fully hidden behind compute.

Core Claim

Under a fixed hardware budget, the scarce resources are bytes moved and fine-resolution tokens computed. Allocating them by the model's own coarse-level reconstruction error should yield representations that transfer better in the low-label regime than spending the same budget uniformly. Uniform spending here means downsampled whole volumes or random full-resolution crops. The selection decision depends on the current model, so it is made only one coarse forward pass before the data is needed. That makes it a genuine latency problem, solvable only if the per-step fetch budget is set by the transfer-to-compute ratio. We also claim this budget is large enough to matter on PCIe Gen4.

What This Builds On, and Where It Departs

Three systems papers supply the mechanism, and each one differs from our setting in a specific way.

CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting keeps cheap selection attributes on the GPU and offloads heavy color attributes to CPU memory. It then fetches only the 0.39–1.06% of primitives visible from the current camera. We reuse its split between "what you need to decide" and "what you need to compute." CLM's selection, however, is exact geometry. Ours is a learned, approximate guess about where training signal lies, so it can be wrong in ways that hurt the representation.

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs keeps a 1–2 bit "scout" copy of the attention cache on the GPU to predict which full-precision entries to prefetch. It shows that the low-bit copy alone loses retrieval ability, while scout-plus-fetch recovers it. Our coarse volume plays the scout's role. The research file raises an objection: a causal scout cannot anticipate a bidirectional masked encoder. That objection does not apply here. We never predict attention. Our scout runs a complete, cheap bidirectional pass at coarse resolution, and its output, a per-region error map, is the selection signal. We also note a point the file leaves implicit. With purely random masking, the masked and visible patches are known before the step, so nothing needs speculation. Latency only becomes a problem once selection depends on the model.

Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer contributes the constraint rather than the method. Its double-buffered prefetching works only when the transfer of the next chunk fits inside the compute time of the current one. We turn that condition into the rule that sets how many full-resolution regions $K$ can be fetched per step:

$$K \le \frac{B \, t_c}{s},$$

where $B$ is the measured effective host-to-GPU bandwidth, $t_c$ is the duration of the coarse forward pass, and $s$ is the size in bytes of one full-resolution region. The form follows directly from requiring transfer to finish before the fine pass begins.

For corpus-scale data that cannot sit in host RAM, Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage shows that direct SSD-to-GPU transfer reaches 32–48 GB/s and beats CPU-mediated offloading by 1.47×. Its lifetime profiler assumes identical iterations. Our model-dependent selection breaks that assumption by construction, so we replace the profile with the per-step error map.

On the learning side, Advancing Radiograph Representation Learning with Masked Record Modeling motivates the work. It finds that reconstructing high-resolution targets from low-resolution masked inputs matters for downstream accuracy, reaching 88.5% CheXpert AUC at 1% labels. Yet it works on downsampled radiographs, and the file notes that its high-resolution targets scale badly with volumetric depth. Two 3D methods define the state of the art we must beat and the scaling wall we must get past. Autoregressive Sequence Modeling for 3D Medical Image Representation reports that long volumetric sequences limit its patch sizes and batch scale. UniMiSS: Universal Medical Self-supervised Learning via Breaking Dimensionality Barrier works through self-distillation, without reconstruction. Neither spends resolution adaptively.

Research Program

The model is a two-level masked autoencoder, a network trained to reconstruct hidden patches of its input. A shared transformer encoder sees a masked coarse token grid of the entire volume, for example a $512\times512\times256$ CT at $4\times$ downsampling. A coarse decoder reconstructs the masked coarse patches. Its per-region error, smoothed over a few steps, becomes the selection score. The top $K$ regions, with a fixed fraction drawn at random to avoid fixating on noise, are then processed by a fine branch. This branch embeds a masked full-resolution sub-volume and conditions on the coarse encoder's tokens for context. In the spirit of MRM, it reconstructs full-resolution voxels.

The full-resolution corpus lives in pinned host memory, or on NVMe for the full corpus. Region fetches are issued asynchronously the moment the error map is ready. $K$ is set by the budget rule above and re-measured online.

Two scientific questions follow. First, does error-guided allocation of full resolution help low-label transfer beyond what any uniform allocation achieves at equal compute? Second, how stale can the selection be? Selection from the current step requires overlap with the coarse pass. Selection from an error map computed one epoch earlier can be prefetched by an ordinary data loader. If the stale version works just as well, the latency problem disappears, and we will report that plainly. Two risks are named in advance. The error signal may concentrate on scanner artifacts, such as metal streaks or field-of-view edges, rather than anatomy. Selective fine reconstruction may also bias the representation away from the large homogeneous organs that segmentation benchmarks reward.

Baselines

The naive reference is a masked autoencoder on whole volumes downsampled until they fit a 24 GB GPU. This is the default choice of practitioners without large clusters, and it is the point any detail-preserving method must beat.

The strongest simple competitor is a masked autoencoder on random full-resolution crops. It is the dominant practice in 3D medical pretraining, needs no offloading, and sees full detail, though never the whole anatomy at once.

The strongest published methods are AR-SSL4M and UniMiSS. AR-SSL4M reports a mean Dice of 0.7088 on the Medical Segmentation Decathlon (MSD) and parity with full supervision at 10% labels. UniMiSS reports 80.97% Dice on BCV abdominal CT segmentation at 40% labels, against 79.97% for 100% supervision.

All in-house baselines use the same encoder size, pretraining data, fine-tuning protocol, and per-method hyperparameter search budget. They also share a matched GPU-hour pretraining budget on one RTX 4090-class GPU with PCIe Gen4, so the comparison is about allocation, not scale. Where we cannot reproduce AR-SSL4M or UniMiSS under matched compute, we cite their reported numbers as external reference points and say so. Beating them means exceeding 80.97% BCV Dice at 40% labels and 0.7088 MSD Dice. It also means matching full supervision at a lower label fraction than 10%, all with one consumer GPU.

Three ablations isolate the key ingredients:

  • Selection signal. The same two-level model with random region selection at identical $K$. This is the decisive ablation for the claim that error guidance, not merely having a fine branch, drives the gain.
  • Freshness of selection. The stale-error-map variant described above.
  • Overlap. A synchronous-fetch variant, which measures how much wall-clock time, and therefore how many pretraining steps within the budget, the overlap buys.

Why not just…?

Why not just train on random full-resolution crops, which already fit in memory? This is the most serious threat. Crops see fine detail and need no bus engineering. We have no evidence in the file that crops underperform for representation learning. The case against them is indirect: localized lesions are diagnosed against surrounding anatomy, and UniMiSS's volume–slice consistency suggests that global-to-local agreement carries signal. The experiments settle this directly by comparing crop-MAE with our method at equal GPU-hours and labels. If crops match us, the proposal reduces to "crops are enough," and that is worth knowing.

Why not just downsample and rely on the encoder to recover detail? MRM's ablations indicate that blurred reconstruction targets cost downstream accuracy. Its gain from high-resolution targets is measured on 2D radiographs, though, and may be smaller in CT, where many organs are large. Our downsampled baseline at matched compute answers this, especially on small-structure tasks such as pancreas and lesion segmentation, where we expect detail to matter most.

Why not just use a standard data loader with prefetching, since one CT volume crosses PCIe in milliseconds? For a fixed or stale selection, this is correct: bandwidth is not the binding constraint when moving a 128 MB volume at 32 GB/s. The latency problem exists only when selection depends on the current model. The stale-map ablation measures whether that dependence matters. If it does not, the systems component becomes unnecessary for host-RAM-resident data, and its value shrinks to the multi-terabyte NVMe case.

Why not just buy an 80 GB GPU? That would reduce the memory constraint, but the fine-token compute budget remains. Even on large GPUs, uniform full-resolution reconstruction of whole volumes is wasteful. We therefore also run the comparison on an A100-class GPU to test whether adaptive allocation still helps when memory is not binding.

Proposed Experiments

Pretraining uses the public CT/MRI collections pooled by AR-SSL4M, its 23,287 volumes plus DeepLesion. Downstream evaluation fine-tunes at 1%, 10%, 40%, and 100% labels on BCV/Synapse multi-organ segmentation, MSD, and BraTS. For BraTS, nnFormer's 86.4% Dice and 4.05 mm HD95 under full supervision (nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer) serve as the fully supervised reference. A secondary 2D test pretrains on MIMIC-CXR at native radiograph resolution and evaluates on CheXpert and SIIM at 1% labels. The reference numbers there are MGCA's 88.8% AUROC and 49.7% Dice from Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning, and MRM's 88.5% AUROC.

The reported metrics are Dice, HD95, and AUROC, each with confidence intervals over fine-tuning seeds. For small structures, we also report Dice stratified by structure size.

On the systems side, we report the fraction of each step the GPU sits idle waiting for transfer, achieved bandwidth against the budget rule, and pretraining throughput relative to an all-in-VRAM run at reduced size.

Three checks decide the claim.

  • Selection value. Error-guided selection must beat random selection at identical $K$ on low-label Dice and AUROC. Without this, the core claim fails.
  • Comparison with uniform spending. The full method must beat both crop-MAE and downsampled MAE at matched GPU-hours, and the gain must concentrate on small structures.
  • Latency. Measured GPU idle time under the budget-derived $K$ must stay near zero, and fresh selection must outperform stale selection. If stale selection is as good, we report that the latency problem dissolves and the contribution is purely the allocation policy.