Deep Supervision That Keeps Tiny Lesions Visible, Using Every Scan in a Sparse Series

Many 3D segmentation networks are trained with deep supervision: extra prediction heads are attached to intermediate, lower-resolution layers of the decoder and scored against a shrunken copy of the ground-truth mask. The heads are discarded after training. For small lesions this creates a problem. When a lesion of a few millilitres is shrunk by a factor of 8 or 16 per axis, it can occupy less than one voxel, and the shrunken mask often contains no lesion at all. The coarse heads are then trained to say "nothing here," and their gradients teach the shared layers to suppress exactly the faint signals that the full-resolution output needs. We propose to fix this in two ways. First, we replace shrunken masks with targets that preserve whether a lesion is present and soften how confident each coarse head is asked to be. Second, we use the extra scans that longitudinal datasets provide only at training time as additional targets for these heads. Clinical cohorts typically have just two or three scans per patient, taken at irregular intervals.

This matters because small lesions are where current models fail. On the ISLES'24 final-infarct prediction challenge, performance drops steeply below 5 mL, and the overall lesion-wise F1 is only 14.94% (ISLES'24). BeSt-LeS benchmarks deeply supervised 2D and 3D U-Nets on stroke lesions and reports a consistent "small lesion struggle." Every change we propose affects training only. The deployed network is an ordinary 3D U-Net with no added parameters, run time, or input requirements.

Core Claim

We claim that much of the small-lesion deficit in deeply supervised 3D segmenters is self-inflicted, and that a better training signal can recover part of it. The deficit has two sources. Coarse-scale targets erase small lesions, and coarse heads are forced to make sharp predictions before their features can support them. Two training-only changes should help, and they can be tested separately.

The first change is presence-preserving, confidence-scheduled intermediate targets. It is meant to stop coarse layers from learning to suppress small lesions.

The second change is time-conditioned auxiliary targets drawn from a patient's other scans. Each coarse head receives the lesion map from each of that patient's other timepoints, together with the elapsed time between scans. This turns a sparse series of two or three scans into extra supervision per patient. At inference, the network outputs only the mask for the target timepoint.

The specific prediction is concrete. Gains should concentrate in lesions under 5 mL. Gains should shrink or vanish for large lesions. The gains should appear with the deployed architecture unchanged.

How This Relates to Prior Work

Three papers supply the mechanism.

Deep Trajectory Supervision shows that classical deep supervision creates an "impedance mismatch" by demanding confident outputs from immature layers. It replaces uniform intermediate targets with an exponential confidence schedule and prunes the heads at inference. However, it is evaluated only on 2D classification, where shrinking the target is not an issue. We carry its schedule over to dense 3D prediction. We argue that in segmentation the mismatch has a second, geometric component that the schedule alone cannot repair: the target itself is wrong at coarse scales.

BackSplit shows that training-only extra output channels can cut false positives on tiny lesions at zero deployment cost. In its case, the extra channels split the background into anatomical structures. It supervises only the final layer and uses single scans. We reuse its train-then-discard principle at intermediate depths and along time, and we treat BackSplit itself as a competitor to beat or combine with.

CRONOS encodes real-valued, irregular timestamps with Fourier features, meaning sinusoids of the elapsed time, to model continuous anatomical change. We borrow only this time encoding, and only for the auxiliary heads. CRONOS optimizes whole-volume image fidelity and has no protection for focal lesions. We do not try to generate images at all.

Two works constrain the design. DB-MSMUNet gets small-lesion gains from permanent extra decoder branches plus deep supervision. Its inference cost is real, and any gain we claim must be measured against architectural capacity of that kind. The cascaded detect-then-segment pipeline in A Unified 2D Framework for DeepLesion shows the cost of a separate detection stage: a missed detection can never be segmented. That is why we keep a single network rather than adding a localization stage.

Research Program

The base model is a 3D U-Net trained in the standard way, with heads at decoder scales $s \in \{1, 2, 4, 8, 16\}$, where $s$ is the downsampling factor per axis.

Presence-preserving targets. For a coarse cell $c$ at scale $s$, let $a_s(c)$ be the fraction of its voxels covered by lesion. This is ordinary average pooling of the binary mask. We renormalize per connected lesion $k$ so that each lesion keeps a peak of 1 regardless of its size:

$$\tilde y_s(c) = \max_k \frac{a_s^{(k)}(c)}{\max_{c'} a_s^{(k)}(c')},$$

where $a_s^{(k)}$ is the coverage contributed by lesion $k$ alone. A lesion smaller than one coarse cell therefore still produces a target of 1 at its cell, instead of vanishing under nearest-neighbour downsampling or being diluted under plain averaging.

Confidence schedule. Following Deep Trajectory Supervision, each head's target is softened toward the dataset's lesion prevalence $\pi$:

$$t_s = g_s \tilde y_s + (1 - g_s)\pi,$$

where $g_s \in (0,1]$ is the confidence assigned to scale $s$. The confidence grows exponentially from the coarsest head to the full-resolution output, using DTS's two parameters $G_{\max}$ and $\beta$.

Temporal auxiliary heads. For a training patient with scans at times $t_1, \dots, t_n$ and target time $t^\*$, each coarse head gets an additional output channel. That channel is conditioned on a Fourier embedding of $t_j - t^\*$ and is supervised with the presence-preserving target built from the lesion mask at $t_j$. This gives up to $n$ targets per patient instead of one. Through the shared decoder, the network learns where small lesions tend to appear or persist over time. All of these channels are deleted at inference.

The analysis has three parts, each tied to one ingredient. First, we will measure how often coarse targets are empty for lesions in each size bin under each target type. This quantifies the erasure hypothesis directly from the labels, before any training. Second, we will probe intermediate features with linear read-outs to test whether coarse layers in standard deep supervision encode small lesions less well than coarse layers trained without deep supervision. That would show active suppression rather than mere under-training. Third, we will measure the angle between auxiliary and primary gradients per size bin. Interference between tasks is a known risk with auxiliary heads, so we need to check whether the temporal heads conflict with the main output.

Baselines

The naive reference points are the same 3D U-Net trained with an output-only loss, and the same network with standard deep supervision using nearest-neighbour downsampled masks. The second is the configuration whose behaviour on small lesions BeSt-LeS documents. Comparing the two shows whether standard deep supervision already hurts small lesions, which is the premise of the proposal.

The strongest training-only competitor is BackSplit. It targets the same small-lesion false-positive problem at zero inference cost. We will train it on the same backbone, using anatomical background labels from the same automated labelling tool it uses. We will also test it combined with our method, since the two are compatible.

The strongest capacity-based competitor is a DB-MSMUNet-style permanent dual decoder adapted to 3D (DB-MSMUNet). We will report its added inference cost alongside its accuracy.

For ISLES'24, the reference figure is the challenge's reported overall lesion-wise F1 of 14.94% (ISLES'24). Because our backbone and preprocessing will differ from the challenge entries, we will treat our reproduced standard-deep-supervision 3D U-Net as the operative baseline rather than claim a direct leaderboard comparison. No reported result was found on Lacuna for small-lesion-binned metrics on the autoPET/CT IV longitudinal data, and none for BeSt-LeS per-size numbers.

Ablations isolate each ingredient. One variant uses presence-preserving targets without the confidence schedule. One uses the schedule without presence preservation, which is plain DTS ported to segmentation. One adds the temporal heads but feeds them a fixed or shuffled time gap, to test whether the time conditioning matters. One gives the temporal heads targets from the same timepoint, which matches the number of heads and gradient paths without adding new temporal information.

Fairness. All methods share the same data splits, preprocessing, number of epochs, and a hyperparameter search of equal size: twelve trials per method on a held-out validation fold, covering loss weights and schedule parameters. Results are reported over five random seeds.

What counts as winning. Beating the baselines means a lesion-wise F1 improvement on lesions under 5 mL that exceeds the spread across seeds, with no significant loss on larger lesions and no increase in inference compute.

Why not just…?

Why not just max-pool the masks in standard deep supervision? This is the most serious threat. Max-pooling the mask also keeps lesions present at coarse scales, and it would plausibly capture most of the first ingredient's benefit. We do not argue around this. Max-pooled targets are a dedicated ablation. If they match our renormalized targets, we will adopt them and credit the gain to presence preservation alone. The proposal's distinct contribution then rests on the temporal heads and the confidence schedule, which max-pooling does not provide. One plausible weakness of max-pooled targets is a different failure: they are hard, fully confident labels at coarse scales, exactly the condition Deep Trajectory Supervision identifies as harmful (Deep Trajectory Supervision). Comparing "max-pool alone" against "max-pool plus schedule" will show whether that matters for segmentation.

Why not just reweight the output loss by lesion size? A loss scored per lesion, or weighted by inverse lesion size at the final layer, addresses class imbalance without touching intermediate layers. It cannot, however, stop coarse heads from teaching suppression if standard deep supervision remains in place. Without any deep supervision, it also gives early layers no direct signal. We will run this as a baseline both alone and combined with our method. If it alone matches our small-lesion gains, our erasure hypothesis is wrong. The feature-probing analysis will show independently whether coarse layers are actually suppressing small lesions.

Why not just feed the other scans as inputs? When a prior scan exists at inference, as in follow-up segmentation on autoPET/CT IV, using it as an input is natural. Promptable LongiSeg and Deep Lesion Tracker exploit paired scans in exactly this way. We will include a two-input baseline in that setting and test whether the temporal auxiliary targets still help on top of it. In forecasting settings like ISLES'24, however, the future scan does not exist at inference. Training-only targets are then the only way to use it, so that is where the temporal ingredient must prove itself.

Why not just model the trajectory generatively, as CRONOS does? A continuous image generator like CRONOS is optimized for voxel-level fidelity across the whole volume. Its own reported limitations include the difficulty of beating the trivial strategy of predicting the most recent scan, and a gap between image-fidelity scores and clinical lesion metrics. We expect it to blur focal lesions. Training a full flow-matching model on the same data is outside our compute budget, so we will not claim to have beaten it. Instead, we will compare against its cheapest strong reference, the "predict the most recent scan" baseline, using lesion metrics.

Proposed Experiments

We will use three testbeds. The first is ISLES'24 final-infarct prediction, from acute CT to the follow-up lesion (ISLES'24 data). This is the forecasting setting. The temporal heads are used only where more than one labelled timepoint is available. The second is autoPET/CT IV longitudinal whole-body CT, the dataset used by Promptable LongiSeg, with baseline and follow-up lesion masks. This is the follow-up setting, and it also tests whether the method transfers beyond the brain. The third is the stroke data benchmarked in BeSt-LeS, which is single-timepoint. There we test the first ingredient alone, with no temporal heads.

Metrics. We report lesion-wise F1 and Dice, binned by lesion volume (under 1 mL, 1 to 5 mL, and above 5 mL), along with false positives per scan and volume-difference error. Inference compute is reported for every model.

Decisive checks. We will run three, in order.

First, the label-level count of erased lesions must show that standard downsampled targets lose a substantial share of lesions under 5 mL at scales 8 and 16. If they do not, the premise fails and we stop.

Second, standard deep supervision must underperform output-only training on lesions under 5 mL while matching or beating it on large lesions. This is the suppression signature. If it does not appear, the erasure story is not the bottleneck. The proposal then reduces to an evaluation of the temporal heads alone.

Third, the full method must beat both the max-pooled-target and size-reweighted baselines on small-lesion F1 across seeds. The shuffled-time-gap ablation must also lose to correct time conditioning on the longitudinal sets. Without that, the temporal ingredient is just additional heads rather than temporal information.