In medical deep learning with fewer than a hundred patients, the typical recipe is a hybrid pipeline. Researchers chain several kinds of model: a signal decomposition feeding a classifier, a diffusion generator feeding a transformer, or a network feeding a classical deformable contour. Each extra stage adds settings to tune, such as decomposition modes, noise schedules, trade-off weights and per-stage learning rates. On a large dataset that is mainly a compute cost. On a cohort of 45 echocardiography sequences it becomes a statistical cost. Every configuration tried is scored on the same few validation patients. Picking the best of many noisy scores inflates the reported accuracy, even if no individual run overfits.
This note proposes to measure that inflation directly for published small-cohort hybrid pipelines. It then tests one remedy: fix the hybrid's extra settings from outside data or domain priors before seeing the target cohort, so the hybrid spends no selection budget on the small data. The claim, in two sentences, is this. Part of the reported advantage of hybrid pipelines over simple models on tiny medical cohorts comes from searching over more settings, not from better modeling. The advantage that remains after equalizing and pre-fixing the search budget is the part worth keeping.
Core Claim
The two research directions meet at a quantity neither tracks, which we call the selection budget. The hybrid-cost literature documents that combining model families "multiplies the hyperparameter space." Examples include the trade-off weight $\lambda$ between physics and network in Hybrid Neural Network Augmented Physics-based Models for Nonlinear Filtering and the "intricate hyperparameter tuning" of the diffusion-plus-polynomial model in Conditional Denoising Meets Polynomial Modeling. That literature treats this as a compute problem.
The small-sample literature, meanwhile, increasingly uses exactly these hybrids as its tool against data scarcity. Examples are the empirical-mode-decomposition recombination in Assessing the Potential of Data Augmentation in EEG Functional Connectivity for Early Detection of Alzheimer's Disease, the diffusion-then-classifier pipeline of DiffMDD, and the CNN–autoencoder–snake pipeline of A Left Ventricular Segmentation Method on 3D Echocardiography Using Deep Learning and Snake. These papers motivate the proposal: they report hybrid gains on cohorts where selection noise is large. They do not report how many configurations were searched.
A standard extreme-value argument gives the scale of the effect. Suppose $K$ configurations have roughly equal true accuracy $p$, each is scored on $n$ validation patients, and each score is an approximately Gaussian estimate with standard deviation $\sigma = \sqrt{p(1-p)/n}$. Then the expected best score exceeds $p$ by roughly
$$ \Delta_{\text{opt}} \approx \sigma \sqrt{2 \ln K}. $$
This is the usual approximation for the expected maximum of $K$ independent Gaussians; correlated configurations shrink the effective $K$. With $n = 20$, $p = 0.8$ and $K = 100$, the gap is about 0.27. That is larger than most gains reported for hybrids. The number is a bound on how bad the problem could be, not a finding. The experiments below measure the actual gap.
Research Program
The work has three parts.
Audit. We re-implement three small-cohort hybrid pipelines from the file:
- the multivariate empirical-mode-decomposition augmentation with ResNet-18, BrainNet CNN and EEGNet classifiers (Jia et al.);
- the diffusion-denoising and augmentation pipeline with a 1D-CNN-Transformer (DiffMDD);
- the CNN, stacked-autoencoder and gradient-vector-flow snake segmentation pipeline on the 45 CETUS 2014 sequences (Dong et al.).
For each pipeline we enumerate every tunable setting across its stages and give it a documented search range. We then run two protocols:
- a "naive" protocol: random search of size $K$, with the best configuration reported on the same validation folds;
- a nested cross-validation protocol: the outer test patients never influence selection.
The gap between the two, measured as a function of $K$, estimates the optimism introduced by tuning. We also record wall-clock cost per configuration. Expensive multi-stage hybrids may be searched less exhaustively in practice, so compute cost and selection optimism may partly offset each other. Measuring this relationship is part of connecting the two directions.
Remedy. We build "pre-committed" versions of each hybrid, whose non-shared settings are frozen before touching the target cohort. Three sources of frozen values are available:
- physiological priors, for example standard EEG frequency bands and a fixed number of decomposition modes;
- values tuned on a separate public dataset from the same modality;
- the physics-weighted regularization idea from augmented physics models, used with a fixed rather than searched $\lambda$.
Only settings shared with the simple baseline, such as learning rate and weight decay, are tuned on the target cohort. They are tuned with the same budget the simple baseline gets.
Reporting. We report the hybrid's advantage after budget matching. We also report it as a curve over $K$, so readers can see where any published claim sits.
Baselines
The do-nothing reference is a single monolithic network with default settings and no search. For classification this is a ResNet-18 on the same inputs. We choose it because Unveiling COVID-19 from Chest X-ray with deep learning found that shallower ResNet-18 beats ResNet-50 on small cohorts because deeper models latch onto hospital- and scanner-specific artifacts. For segmentation the reference is a plain 3D U-Net. This point shows what zero selection budget buys.
The budget-matched simple baseline is the same monolithic network tuned by random search with the same $K$ and the same nested cross-validation as the hybrid. Without it, the comparison confounds modeling with tuning.
The strongest existing methods are the three published hybrids, run with their full search space under nested cross-validation. The file does not give their reported numbers:
- no reported result found on Lacuna for the CETUS 2014 Dice score of Dong et al.;
- no reported result found on Lacuna for DiffMDD's MDD classification accuracy;
- no reported result found on Lacuna for the Alzheimer's/MCI accuracy of Jia et al.
We will extract these numbers from the original papers before running anything. The first check is whether our naive-protocol reproduction reaches them.
A pretrained-representation baseline is a frozen pretrained encoder with a linear probe. This follows the frozen-backbone approach described in Towards AI-driven Next Generation Personalized Healthcare and Well-being, which uses a pretrained speech model without retraining it on small clinical sets. This baseline has almost no selection budget and is the most serious competitor.
The key ablation is the pre-committed hybrid against the same hybrid tuned on the target cohort at equal $K$. It isolates whether freezing hybrid-specific settings recovers honest performance.
Beating the baselines means the following. The pre-committed hybrid must exceed the budget-matched simple baseline and the frozen-encoder baseline on outer-fold patients. For classification the margin must be larger than the outer-fold standard error; for segmentation it is judged on Dice. The hybrid must also show a smaller naive-versus-nested gap than the fully tuned hybrid.
Why not just…?
Why not just use nested cross-validation and stop? Nested cross-validation fixes the measurement, and we use it everywhere. It does not make a hybrid better; it only reveals whether the hybrid was better. If the audit shows hybrids lose their edge under nesting, the result is still useful but negative. The pre-committed variant is the constructive part, and the key ablation tests whether it adds anything beyond honest evaluation.
Why not just use a smarter search, such as Bayesian optimization, to cut the cost? Cheaper search reduces the compute cost the hybrid-cost literature worries about. It does not reduce selection optimism. A search that concentrates on the most promising configurations may even increase it, because it exploits validation noise more efficiently. We include Bayesian optimization as a search variant and report its naive-versus-nested gap alongside random search.
Why not just use a frozen pretrained encoder and skip hybrids entirely? This is a real threat. Pretrained representations transfer prior knowledge into the model without spending selection budget on the target cohort, which is what pre-commitment tries to do. If the frozen-encoder baseline matches the pre-committed hybrids, the honest conclusion is that hybrids are not worth their cost on small cohorts. The comparison will settle this per modality. Snake-style geometric refinement on 3D echo may keep an edge that EEG classification does not.
Proposed Experiments
The decisive experiment is the gap-versus-$K$ curve on each of the three pipelines. We use $K$ from 1 to 200 and 20 repeated outer splits, with the same patients, folds and GPU budget for every method. If the naive-protocol gap grows roughly like $\sqrt{\ln K}$ and reaches the size of published hybrid gains, the audit claim holds. If the gap stays flat, the claim fails and hybrid gains are genuine.
The second check compares pre-committed hybrids with the budget-matched ResNet-18 or U-Net and with the frozen-encoder baseline, all on outer folds.
The third check tests transfer. It applies settings frozen on one EEG cohort to the other, from the Alzheimer's data to the MDD data. This shows whether pre-commitment carries across cohorts or is itself a hidden form of tuning.
All runs log wall-clock time per configuration. This lets us report, for each pipeline, accuracy per GPU-hour together with accuracy per unit of selection budget.
References
- Hybrid Neural Network Augmented Physics-based Models for Nonlinear Filtering
- Conditional Denoising Meets Polynomial Modeling: A Flexible Decoupled Framework for Time Series Forecasting
- Assessing the Potential of Data Augmentation in EEG Functional Connectivity for Early Detection of Alzheimer's Disease
- DiffMDD: A Diffusion-Based Deep Learning Framework for MDD Diagnosis Using EEG
- A Left Ventricular Segmentation Method on 3D Echocardiography Using Deep Learning and Snake
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Towards AI-driven Next Generation Personalized Healthcare and Well-being