When a clinical dataset has fewer than a hundred patients and only a handful of positive cases, automated network design stops working. Evolutionary architecture search is a method that breeds a population of candidate networks and keeps the ones that score best on a held-out validation split. With five minority-class examples in that split, one misclassified patient swings the score enough to reorder the whole population. The search then selects noise. We propose to fix the score rather than the search. Each candidate network is judged by what its own learned features can do once the rare class is densified. Concretely, we fill in the rare class inside each candidate's feature space with simplicial over-sampling, which creates synthetic points inside triangles and tetrahedra spanned by neighbouring real examples rather than only along lines between pairs. We then fit a cheap linear classifier on top, and average the resulting score over many stratified resplits.
The claim is specific and falsifiable. Under combined sample scarcity and class imbalance, this "densified-head" fitness will track true generalization far better than the usual single-split validation score, where tracking is measured by rank correlation with performance on a large untouched test set. Better tracking should in turn let evolutionary search beat a well-tuned ResNet-18, the fixed network that has embarrassed complex automated pipelines on small medical data. If the search cannot beat that network even with a trustworthy score, we will report it as the answer to whether architecture search is worth running at N < 100.
What the Literature Establishes and Leaves Open
Three findings motivate the proposal. IMB-NAS trained 135 architectures on balanced and imbalanced CIFAR. It showed that architecture rankings decorrelate (low Kendall's tau) under imbalance, and that retraining only the linear head of a balanced backbone with delayed re-weighting matches a full search from scratch (41.93% vs. 41.53% on CIFAR-100 at 100x imbalance). USAA evolves augmentation policies and architectures jointly on MedMNIST with AUC as fitness. It identifies "in-domain sampling bias": small train and validation splits differ in distribution, so validation scores fluctuate. It also finds that small search spaces beat large ones on small data. The MedMNIST decathlon and MedMNIST v2 report that AutoKeras overfits small datasets and that plain ResNet-18 often beats deeper models and complex automated systems.
These papers disagree in a way no one has resolved. IMB-NAS says the head is where imbalance should be handled. USAA says joint search still pays off on small medical data. Neither tests the regime where both problems hold at once, and neither measures whether the search's own fitness signal is trustworthy there.
Three papers supply the method.
- Simplicial SMOTE. Simplicial SMOTE finds cliques in a nearest-neighbour graph of minority points and samples inside the resulting simplices with Dirichlet weights. It reports average F1 gains of 4.5% (k-NN classifier) and 5.0% (gradient boosting) over SMOTE on 21 tabular datasets. It is tabular only and never touches learned features.
- DeepSMOTE. DeepSMOTE interpolates in the latent space of a fixed encoder-decoder and decodes back to images. We keep its insight that interpolation belongs in feature space, but drop the decoder, because a decoder cannot be trained from ten minority images. We also replace its one-dimensional line interpolation with simplices.
- FreMGP. FreMGP uses genetic programming to evolve synthetic time-series samples for fixed classifiers. We do the reverse: the synthetic data generator is simple and fixed in form, and evolution acts on the architecture.
Core Mechanism
Each individual's chromosome encodes three things:
- a compact convolutional topology, using a stage-wise binary connectivity encoding as in Genetic CNN, inside a deliberately small search space following USAA;
- the maximum simplex dimension $p$;
- the minority-to-majority target ratio for over-sampling.
Fitness evaluation has five steps.
- Train the backbone briefly on the training fold, using class-balanced mini-batches.
- Extract penultimate-layer features for all training examples.
- Generate synthetic minority features as $\tilde z = \sum_{i=1}^{q} w_i z_i$ with $w \sim \mathrm{Dirichlet}(1,\dots,1)$. Here the $z_i$ are features of $q \le p+1$ minority examples forming a clique in the class's k-NN graph.
- Fit a logistic-regression head on real plus synthetic features.
- Score the head on the validation fold.
Because only the head is refit, steps 3–5 are cheap. This lets us repeat the whole procedure over $K$ stratified resplits. We then use
$$F(a) = \frac{1}{K}\sum_{k=1}^{K} \mathrm{AUC}_k(a) - \lambda\, \mathrm{sd}_k\big(\mathrm{AUC}_k(a)\big),$$
where $a$ is the candidate, $\mathrm{AUC}_k$ is the validation area under the ROC curve on resplit $k$, and $\lambda \ge 0$ penalizes candidates whose score depends on the split. Selection, crossover and mutation follow standard tournament-based evolutionary search, as in Sun et al..
Minority classes with very few points yield few large cliques. The chromosome's $p$ therefore falls back automatically to edges ($p=1$, plain SMOTE) when higher cliques do not exist. How often this fallback happens is itself a reported quantity.
Research Program
Constructing the regimes. We build controlled small-sample regimes by subsampling MedMNIST training sets to N ∈ {50, 100, 200} with imbalance ratios ∈ {5, 20}. Every model is scored on the full official MedMNIST test set. This is the key design choice: it gives a nearly noise-free ground-truth ranking of architectures even when training data are tiny, which real private cohorts never provide. Datasets are BreastMNIST and PneumoniaMNIST (binary), DermaMNIST (seven classes, heavily skewed), and RetinaMNIST (ordinal, small). OrganMNIST3D serves as a 3D check.
Fitness-reliability study. For a fixed pool of roughly 150 sampled architectures, we compute Kendall's tau between each fitness variant and full-test AUC, across 20 random draws of the small training set. The variants are:
- single-split cross-entropy validation AUC, as in USAA-style searches;
- repeated-split AUC;
- repeated-split AUC with a delayed-re-weighted head (the IMB-NAS P1 recipe);
- repeated-split AUC with a 1-D SMOTE head;
- the full densified-head fitness.
Search. We run evolutionary search under each fitness variant at a matched GPU-hour budget. The resulting architectures are compared against the baselines below.
Baselines
The naive reference point is ResNet-18 at 28×28, trained with cross-entropy on the subsampled set and tuned with the same number of trials the search consumes. This is the configuration MedMNIST v2 found hard to beat at low resolution. We add a variant with ImageNet-pretrained weights. The proposal must beat this on full-test AUC, averaged over the 20 data draws, with a paired test.
The strongest head-level competitor, and the one we consider the real threat, is ResNet-18 with the same densified head (simplicial over-sampling in penultimate features, then logistic regression) and no architecture search. It isolates whether any gain comes from the search or merely from the head. Beating it by a statistically reliable margin, not just matching it, is the bar for claiming search helps.
Loss-based imbalance handling is represented by ResNet-18 with logit adjustment or focal loss, following the fine-tuning recipe of Timofeev et al.'s self-supervised NAS, and by IMB-NAS's frozen-backbone delayed re-weighting. All are tuned with the same trial budget.
The strongest existing search method is a USAA-style evolutionary search over the same small space, with single-split AUC fitness and standard augmentation. USAA reports 1.2% higher average AUC than Google AutoML Vision on MedMNIST, and MedMNIST v2 reports that Google AutoML Vision reaches 0.927 average AUC on the 2D tasks with full training data. No reported result was found on Lacuna for any method in the N < 100 subsampled regime, so we rerun USAA-style search ourselves at matched compute rather than quoting numbers.
Ablations swap each ingredient of the fitness while holding everything else fixed:
- delayed re-weighting in place of over-sampling;
- lines ($p=1$) in place of simplices;
- $K=1$ in place of repeated splits;
- $\lambda=0$ in place of a variance penalty.
Together these show which ingredient restores rank correlation.
Why not just…?
Why not just tune a ResNet-18 and skip search? This might be the right answer. MedMNIST v2 and the MedMNIST decathlon both show fixed small networks outperforming complex automated pipelines on small data. Our comparison against the densified-head ResNet-18 at matched compute settles the question directly. If search does not beat it, the fitness-reliability study still stands as a contribution: it would show that the failure lies in the noisy score rather than in search as a whole.
Why not just re-weight the loss instead of over-sampling? IMB-NAS shows head-level re-weighting is enough on CIFAR. That evidence comes from thousands of samples per head class, though, and re-weighting cannot add support where the minority class has five points. The ablation with a delayed-re-weighted head tests this directly. If it matches simplicial over-sampling on rank correlation and final AUC, we will drop over-sampling from the method.
Why not just over-sample pixels or decode latent samples, as DeepSMOTE does? DeepSMOTE needs to train an encoder-decoder, which is implausible from tens of images. Pixel-level SMOTE interpolates across unrealistic image mixtures. Feature-space sampling feeding a linear head avoids both problems. We include a pixel-SMOTE training baseline to show the difference.
Why not just use repeated resplits without any over-sampling? Averaging reduces variance but does not change what is measured: a head trained on five minority points. The ablation with repeated splits alone separates variance reduction from densification. We expect both to matter, but this is exactly the variant that could make over-sampling unnecessary, and we will say so if it does.
Decisive Checks
The central experiment is the rank-correlation study. The proposal succeeds only if densified-head fitness achieves a clearly higher Kendall's tau with full-test AUC than single-split fitness at N ≤ 100, across draws and datasets. It fails if this gap vanishes, or if the ablations show resplitting alone accounts for it.
The second check is the final comparison against densified-head ResNet-18 at equal GPU-hours. We also report parameter counts of the evolved networks as N grows, to test USAA's conjecture that small data favours small capacity even after synthetic densification. A positive result would give practitioners a search procedure whose score can be trusted on tiny cohorts. A negative one would give clear evidence that, below about a hundred patients, a fixed small network with a densified head is all they need.
References
- IMB-NAS: Neural Architecture Search for Imbalanced Datasets
- Auto Machine Learning for Medical Image Analysis by Unifying the Search on Data Augmentation and Neural Architecture
- MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis
- MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification
- Simplicial SMOTE: Oversampling Solution to the Imbalanced Learning Problem
- DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data
- Contrastive Representation-Guided Genetic Minority Oversampling for Imbalanced Time-Series Classification
- Genetic CNN
- Evolutionary Neural Architecture Search for Multivariate Time Series Forecasting
- Self-Supervised Neural Architecture Search for Imbalanced Datasets