Do Dense Connections Carry Shortcuts Forward in Small Medical Datasets?

Densely connected networks feed every layer's output forward to all later layers by stacking the feature maps together rather than adding them. This design is valued because fine early detail survives to the deep layers. Small medical datasets push models toward the opposite of what we want: they latch onto early, simple cues such as scanner markers, burned-in text, or collimator edges instead of the disease itself. These two facts collide. If simple shortcuts are learned in the earliest layers, and dense wiring hands those layers straight to the classifier, then dense networks may become more shortcut-prone as the cohort shrinks. Depth can no longer wash the artifact out.

We propose to test this directly and to exploit a property unique to dense networks. Because each layer reads a stacked input, the small mixing convolution at the front of each layer (a $1\times1$ convolution that compresses the stack) has weights that are literally partitioned by source layer. We can read off how much each deep layer relies on each shallow layer. This gives a shortcut diagnostic that needs no clean reference model. It also suggests a targeted fix: penalize how strongly deep layers read from the shallowest sources. The note asks whether that fix beats the cheap alternatives a reviewer would reach for first. If it does not, we report that honestly.

Core Claim

We make two predictions and offer one tool. The first prediction is that the harm a simple synthetic shortcut does to a dense network grows faster as training-set size falls than it does for a parameter-matched residual network. The second is that this extra harm is carried by the dense paths from early layers into late layers. The tool is a "source reliance" score, defined as the share of fusion-weight norm that a deep layer places on the first $k$ source layers. We expect this score to rise when a shortcut is present, without access to a clean model.

Formally, let $W_l$ be the $1\times1$ fusion weights of layer $l$, and let $W_l^{(s)}$ be the slice acting on channels contributed by source layer $s$. We define

$$R_l^{(k)} = \frac{\sum_{s<k} \|W_l^{(s)}\|_F^2}{\sum_{s<l} \|W_l^{(s)}\|_F^2},$$

where $\|\cdot\|_F$ is the Frobenius norm. Because this raw ratio depends on the scale of each source's activations, we normalize each slice by the batch-norm scale of its source.

The fix adds $\lambda \sum_{l \ge L_d} R_l^{(k)}$ to the training loss. Here $L_d$ marks where the "deep" layers begin and $\lambda$ sets the strength of the penalty. The penalty does not cut the early paths. It makes the deep layers pay to use them, so fine detail stays reachable when the labels genuinely reward it.

Where This Sits

The motivating evidence comes from two papers in the small-data literature. There Are No Shortcuts to Anywhere Worth Going profiles DenseNet-121, ResNet-34, VGG16 and Inception v3 under synthetic shortcuts on CheXpert, ISIC 2017 and Waterbirds. It finds that simple shortcuts settle in early layers and cut AUC by 10–15%. It localizes them with layer-wise KL divergence against a clean model, and it proposes no mitigation. We differ in three ways: we vary sample size systematically, we compare dense and residual wiring under matched parameters, and we replace the clean-model requirement with a weight-based score. Unveiling COVID-19 from Chest X-ray with deep learning shows that models trained on 447 radiographs learn collection artifacts and that ResNet-50 overfits relative to ResNet-18. It supplies both a realistic test bed and the capacity objection we must answer.

Other papers complicate the picture. In prostate MRI, Exploring UNet-based models for prostate lesion segmentation finds that DenseUNet beats a plain U-Net on aggregate Dice (0.57 vs about 0.44) on 222 patients. In breast biopsy segmentation, Learning to Segment Breast Biopsy Whole Slide Images reports that dense decoding with direct raw-input feeding reaches 0.700 pixel accuracy, against 0.597 for a residual encoder-decoder, with only 58 annotated regions. Neither paper tested out-of-distribution data. Read naively, both say dense reuse helps in small data, which seems to contradict our hypothesis. Our reading is that the two claims are compatible. Early-feature reuse helps in-distribution precisely because it gives the head easy access to low-level cues, and that same access is dangerous once those cues are spurious. The experiments separate the two effects by measuring in-distribution accuracy and shortcut-reversed accuracy together.

The method's ingredients come from dense design itself. Residual Dense Network for Image Super-Resolution introduced local $1\times1$ fusion over stacked features, and that fusion is the object we read and penalize. Constraint-based small-data work offers an analogy rather than a method. Deep Residual Dense U-Net adds a Fourier-domain loss, and MFM-Geom adds Riemannian heads. Both constrain outputs or the classification head. Neither constrains which internal layers may talk to which, and that is the gap we target.

Research Program

The study has three parts.

The first part is a controlled sample-size sweep. On CheXpert pneumothorax and ISIC 2017, we inject the synthetic shortcuts from the shortcuts paper: a constant-location square, a random-location square, and a complex shape. Each is correlated with the label at 90% in training and reversed at test. We subsample the training data to sizes of roughly 100, 250, 500, 1000 and 2000. We train DenseNet-121 and a ResNet whose parameter count matches it, five seeds each. The central quantity is the shortcut gap, meaning AUC on a clean test set minus AUC on the reversed test set, plotted against training-set size for each architecture.

The second part is localization. At each size we compute $R_l^{(k)}$ and run targeted ablations, zeroing the early-source slices in deep fusion layers at test time. This tells us whether the gap is actually carried by those paths. We also compare $R$ against the KL-based localization from the shortcuts paper wherever a clean model exists. That comparison validates the clean-model-free score before we use it alone.

The third part is the intervention and real-world transfer. We train with the reliance penalty, tuning $k$ and $\lambda$ on a validation split that has no shortcut reversal. This keeps the deployment setting honest, since we would not know the shortcut in advance. For a realistic test, we use the CORDA chest X-ray protocol and its public code, eidoslab/unveiling-covid19-from-cxr, training on public COVID data and testing on CORDA. There the shortcut is unknown and only cross-dataset performance can tell.

Baselines

The do-nothing reference is DenseNet-121 trained conventionally, with ImageNet initialization, the same augmentation, and a learning rate tuned per sample size. Every other result is reported as a difference from it.

The capacity baselines are ResNet-18 and a parameter-matched ResNet-34. They are the strongest competitors on the "denser versus smaller" question because the COVID chest X-ray paper found the shallower ResNet-18 more robust in small data. Its reported CORDA result is sensitivity 0.54 and specificity 0.80, with RSNA pretraining and lung segmentation. On CORDA we reproduce exactly that pipeline. Beating it means matching or exceeding both numbers when the dense network is run under the identical pipeline.

The lowered-learning-rate baseline follows the shortcuts paper. That paper reports that dropping the learning rate from 0.01 to 0.001 improves AUC by about 8% but leaves a 3–11% shortcut degradation. It is the only mitigation in the file with a reported effect. No absolute per-architecture CheXpert AUC is reported on Lacuna, so we re-run it ourselves. Beating it means a smaller reversed-test shortcut gap at equal clean AUC.

The ablations isolate our ingredient. Truncated dense wiring removes early-source inputs to deep layers entirely. Stochastic source dropout randomly zeroes whole source slices during training. Both receive the same tuning budget as our penalty.

All methods share data splits, augmentation, and a budget of 20 tuning trials and five seeds. They are also matched in parameters within 5%.

Why not just…?

Why not just use a smaller residual network? Shrinking the model may reduce shortcut harm without any dense-specific mechanism. Against this, the prostate and biopsy papers show that plain and residual networks lose substantial in-distribution accuracy relative to dense ones in small data. The sweep reports both axes at once, clean accuracy and shortcut gap. If ResNet-18 dominates the dense penalty on both, the practical conclusion is "use ResNet-18," and we will say so.

Why not just lower the learning rate? It is cheap and reportedly recovers about 8% AUC. But that same paper shows a residual 3–11% shortcut degradation, so it does not finish the job. We also test whether lowering the learning rate and applying the penalty combine additively.

Why not just cut or randomly drop the early paths? This is the real threat. Truncation and source dropout are simpler than a penalty. If either matches it, the methodological contribution collapses to the diagnostic and the sample-size law. Truncation should cost clean accuracy on fine-boundary tasks, because it removes what dense reuse is good for. Dropout may not cost accuracy, but it also cannot prefer one source over another, whereas the penalty prices them differently. The ablation decides this, and we treat a tie as a result rather than something to argue away.

Why not just pretrain contrastively or augment? Contrastive Pretraining for Echocardiography Segmentation reaches Dice 0.9125 with 5% of labels, against 0.9204 for full supervision. However, the COVID chest X-ray results show that pretraining and augmentation do not remove collection biases under cross-dataset testing. We add a SimCLR-pretrained DenseNet arm on unlabeled CheXpert images to test whether pretraining shrinks the dense-specific gap on its own.

Decisive Checks

The hypothesis fails if the dense and residual shortcut-gap curves run parallel as training-set size falls. It also fails if zeroing early-source slices does not shrink the gap. The diagnostic fails if $R_l^{(k)}$ does not rise under shortcuts when no clean model is used. The fix fails if it does not beat dropout and truncation on reversed-test AUC while staying within one point of clean AUC. The final test is transfer: on CORDA, which has no synthetic shortcut, the dense network with the penalty should match or beat the reported ResNet-18 sensitivity and specificity. If the controlled gains do not show up there, they are artifacts of the synthetic setting.