When a hospital has only a few dozen labeled scans, the usual move is to take a network pretrained on natural images and fine-tune it. This works on the hospital's own test split and then often fails at a second hospital. The model has learned the scanner, the text tags burned into the image, or the site's contrast profile instead of the disease. We propose to rewrite each pretrained convolution, exactly and without retraining, as a sum of two simpler operations: per-channel spatial filters, and a separate step that mixes channels at each pixel. We then fine-tune only one of those two parts. The spatial filters carry the shape and texture detectors learned from millions of images. The channel-mixing weights decide which detectors matter for the new task. Our bet is that training only the channel mixing, with the pretrained spatial filters frozen, gives a constrained way to adapt. It should overfit less and transfer better to an unseen hospital than full fine-tuning or a generic low-parameter update of the same size.
The building block is the depthwise separable convolution, a layer that first filters each input channel spatially on its own and then mixes channels with a $1\times1$ convolution. It is usually adopted for speed: roughly a ninefold parameter cut for $3\times3$ kernels, as popularized by Xception. The new element here is to use the factorization as a statement about which parameters may move when data are scarce, not as a compression trick. The proposal stands or falls on one measurement. Adapting the channel factor alone must beat adapting an equally sized but arbitrary set of parameters on external-cohort performance.
Core Claim
Network Decoupling shows that a standard $K\times K$ convolution can be expanded by singular value decomposition into a sum of depthwise separable components. With all $T=K^2$ components kept, the expansion is exact. The paper also finds that a pointwise-then-depthwise ordering concentrates the weight energy into the leading terms. Concretely, a pretrained weight $W$ becomes
$$W = \sum_{t=1}^{T} D_t \circ P_t,$$
where $P_t$ is a $1\times1$ channel-mixing map and $D_t$ is a set of per-channel $K\times K$ spatial filters. Network Decoupling and the related GDWS use this expansion to speed up inference on ImageNet, CIFAR and PASCAL VOC with large training sets. Neither asks what happens when you train through the factors on a hundred images.
We claim that freezing every $D_t$ and fine-tuning only the $P_t$ (together with normalization parameters and the head) is a better inductive bias for small cohorts than either full fine-tuning or parameter-matched generic updates. The test is external-site specificity and AUC. A secondary claim is the mirror image: freezing $P_t$ and adapting $D_t$ should help when the target differs from the source mainly in spatial structure. The ratio of the two outcomes is a diagnostic of where a domain gap lives.
Because the expansion is exact and linear, no nonlinearity is placed between the factors. This sidesteps the degradation Xception reports when ReLUs are inserted between depthwise and pointwise stages.
Research Program
The work has three parts.
Decomposition tool. We will build a decomposition tool for ResNet-18. It is the backbone that the COVID-19 chest X-ray study found overfits less than ResNet-50 on small data. The tool will also cover a 2D/3D variant for MRI. It converts every $3\times3$ convolution into its exact pointwise-then-depthwise expansion and exposes each factor family as a separately trainable group. Network Decoupling reports a 1.4% accuracy drop and only 1.5x speedup when truncating ResNet-18. We therefore treat truncation rank $T$ as a separate axis. Full rank is used for the main claim, and truncated rank serves as an additional capacity restriction.
Adaptation sweep. We will run controlled adaptation sweeps at $N \in \{10, 50, 100\}$ labeled training subjects per class, plus the full training set. We will use subject-level splits and ten random draws per $N$. This matters because the OASIS cohort in the Alzheimer's depthwise separable CNN paper was augmented from 430 subjects to 1,390 images. Image-level splits would leak subjects across folds.
Shortcut analysis. We will measure whether channel-only adaptation reduces reliance on site. The approach follows the bias analysis of the COVID-19 study, which used t-SNE to show features clustering by scanner and institution. We train a linear probe to predict acquisition site from frozen penultimate features and report its accuracy under each adaptation scheme. If our hypothesis is right, channel-only adaptation should yield lower site decodability at matched in-domain accuracy.
Baselines
The naive reference is a depthwise separable network trained from scratch. The Alzheimer's DSC paper reports 77.79% OASIS accuracy for this configuration, against 78.02% for a standard CNN. We rerun it on our subject-level splits, because its augmented split protocol differs.
The strongest in-domain competitor is full fine-tuning of an ImageNet-pretrained network. The same paper reports 93.02% average accuracy on OASIS for fine-tuned GoogLeNet. For external transfer, it reports 75.32% accuracy, 80.13% sensitivity, 65.32% specificity and 81.41% AUC on ADNI MCI-vs-AD. These external numbers are the target to beat. Our proposal counts as beating this baseline if it reaches matching OASIS accuracy within run-to-run noise and improves ADNI specificity and AUC across the ten draws. On chest X-rays, the reference is RSNA-pretrained fine-tuning on CORDA, which the COVID-19 study reports at sensitivity 0.54 and specificity 0.80. We run all fine-tuning baselines on our own ResNet-18 with the same data, augmentation, epoch budget, and equal hyperparameter search trials per method. Paper numbers are used for orientation only.
The most important baselines are the parameter-matched controls, because they isolate the key ingredient. The first is a generic low-rank additive update to each original dense kernel, with rank chosen so that its trainable parameter count equals that of the $P_t$ family. The second trains a random subset of dense-kernel weights of the same size. The third trains normalization layers plus the head only, a common remedy for site shift. The final ablation reverses the roles, training $D_t$ with $P_t$ frozen.
If channel-only adaptation does not beat the low-rank and random-subset controls on external AUC, the factor structure adds nothing beyond parameter counting. In that case we will report it as such.
Why not just…?
Why not just use a shallow ResNet-18 with weight decay, dropout and heavy augmentation? This is the default, and the COVID-19 study supports it over ResNet-50. It controls capacity but does nothing to steer which features change, so site shortcuts remain learnable. It is included as the full fine-tuning baseline under the same regularization. Our method is applied on top of the same ResNet-18, so any gain is attributable to which parameters move.
Why not just use a standard parameter-efficient update such as a low-rank adapter, or tune normalization only? This is the real threat. A low-rank update may restrict capacity just as well, and normalization tuning targets intensity shift directly. The file contains no reported result for low-rank adapters on OASIS→ADNI or CORDA. Rather than argue the point, we make both methods parameter-matched baselines. The proposal is only worth pursuing if the separable factorization wins at equal budget. If normalization-only tuning wins on external specificity, that would show the gap is mostly intensity statistics, which is itself a useful finding.
Why not just crop to the anatomy, as with lung segmentation? The COVID-19 study shows that lung-field masking removes tags and leads, and it releases code for this (EIDOSLAB/unveiling-covid19-from-cxr). Cropping removes shortcuts outside the organ but not scanner texture inside it. We apply cropping to every method in the chest X-ray experiments. Any remaining gain is therefore beyond what preprocessing removes.
Proposed Experiments
The primary experiment trains on OASIS at each $N$ and tests on ADNI MCI-vs-AD. The metrics are AUC, specificity at the in-domain sensitivity threshold, and accuracy, each with confidence intervals across draws. The decisive check is the gap between in-domain and external AUC: channel-only adaptation should shrink it relative to both full fine-tuning and the parameter-matched low-rank control.
The second experiment uses CORDA (447 chest X-rays) with RSNA pretraining, lung cropping, and external testing on public COVID chest X-ray data. It reports sensitivity, specificity and site-probe accuracy.
As a stress test of the reverse hypothesis, we apply the 3D variant to BraTS 2025 pediatric glioma segmentation. Here the nnU-Net enhancement paper reports a lesion-wise Dice of 0.759 on the scarce cystic component, against 0.967 on edema. We ask whether spatial-only adaptation helps that rare, morphologically variable class more than channel-only adaptation does.
A clean negative result would also settle the question. Such a result would be no advantage over parameter-matched controls at any $N$, with equal site decodability across schemes. It would show that, for small cohorts, depthwise factorization is purely an efficiency tool and not a regularizer.
References
- Xception: Deep Learning with Depthwise Separable Convolutions
- Network Decoupling: From Regular to Depthwise Separable Convolutions
- Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Alzheimer's disease detection using depthwise separable convolutional neural networks
- Towards Reliable Pediatric Brain Tumor Segmentation: Task-Specific nnU-Net Enhancements