When a pretrained medical model is adapted with only a few percent of its data labeled, the rare diseases that matter most may have three or four positive examples in the validation set. Hyperparameter search then ranks candidate configurations by a score that can swing wildly when a single prediction flips. The search learns from that noise: it concentrates on configurations that got lucky on a few patients, reports an inflated validation score, and can hand back a model that does worse on held-out patients than an untuned default. This proposal measures that failure directly and fixes it inside the Tree-structured Parzen Estimator, or TPE. TPE is a Bayesian hyperparameter optimizer that sorts past trials into "good" and "bad" groups by validation score, fits a density to each group, and proposes new settings that look like the good group.
The fix is to stop making that sort a hard cut. All trials are scored on the same small validation set, so the question that matters is not how noisy each score is on its own. It is how confident we are that trial A really outranks trial B. We estimate this with a paired bootstrap: we resample validation patients once and rescore every trial on the same resample. Each trial then joins the good group in proportion to its probability of truly being in the top fraction. The same machinery gives a probability that a clinical floor, such as specificity of at least 95%, is truly met. We test this on 12-lead ECG fine-tuning under label scarcity and class imbalance, where the harm from tuning on noise should be largest.
Core Claim
Under label scarcity with class imbalance, standard hyperparameter optimization overfits the validation set. The overfitting grows with the number of trials and shrinks with the number of minority-class validation positives. Two changes to TPE close most of the resulting gap between the configuration that was selected and the best configuration that was evaluated. The first is to weight each trial's membership in the good and bad densities by its paired-bootstrap probability of truly ranking in the top quantile. The second is to pick the final model by a lower confidence bound rather than by the maximum observed score. A plug-in fix that simply replaces each trial's score with its bootstrap mean cannot achieve the same effect. Because every trial shares one validation set, averaging over resamples barely changes the ranking. What noise damages is the ranking itself, and only a ranking-level treatment addresses it.
Concretely, let trial $i$ have hyperparameters $x_i$, and let $y^*$ be the top-$\gamma$ quantile threshold that TPE uses. Over $B$ joint bootstrap resamples of the validation patients, we rescore every trial and compute
$$w_i = \frac{1}{B}\sum_{b=1}^{B} \mathbf{1}\left[y_i^{(b)} \ge y^{*(b)}\right], \qquad \ell(x) \propto \sum_i w_i K(x, x_i), \quad g(x) \propto \sum_i (1-w_i) K(x, x_i).$$
Here $y_i^{(b)}$ is trial $i$'s score, for example macro AUC over rare classes, on resample $b$. The threshold $y^{*(b)}$ is the top-$\gamma$ quantile on that same resample. $K$ is TPE's Parzen kernel. Proposals still maximize $\ell(x)/g(x)$. For a clinical constraint we compute the analogous probability that trial $i$ satisfies the floor, and we use it in place of the hard feasible/infeasible label in the constrained TPE density ratio.
Where This Sits
Three strands of prior work motivate the problem. The TPE formulation in Tree Structured Parzen Estimators for Hyperparameter Optimization splits trials at a quantile of an observed score and implicitly treats that score as exact. Needle in a Haystack develops adaptive importance sampling to evaluate a single fixed model under extreme imbalance. It shows that passive estimates of F1 and precision-recall under tight label budgets have order-of-magnitude higher error than adaptive ones. That paper only evaluates one model. It never asks what happens when an optimizer selects among hundreds of noisy evaluations, and that selection problem is our question. The SCL-TPE work (Supervised contrastive representation learning with TPE Bayesian optimization for imbalanced tabular data) tunes a contrastive temperature with TPE on KEEL/UCI tables. Some of those tables, such as Lymphography and Glass, have minority classes with one or two examples. It reports winning numbers from exactly the kind of selection we suspect is optimistic. We treat SCL-TPE as the closest existing method. It differs from this proposal in three ways: its data are fully labeled tables, it tunes a single scalar, and it uses a hard, noise-blind split.
Two TPE extensions supply methods we reuse. c-TPE adds inequality constraints through relative density ratios, but it was only tested on constraints that can be measured exactly, like memory and latency. A sensitivity floor measured on four positives is a noisy constraint, and c-TPE's feasibility labels would then be wrong. MOTPE supplies weighted membership in the good density: it weights by hypervolume contribution, whereas we weight by rank probability. It also documents that TPE variants degrade on deceptive landscapes. A landscape made deceptive by evaluation noise is the case we target.
The application setting comes from ECG self-supervision. Self-supervised representation learning from 12-lead ECG data shows that contrastive predictive coding (CPC) pretraining helps most on rare PTB-XL classes. It reaches a fine-tuned macro AUC of 0.9418 against 0.9318 for supervised training, but its fine-tuning recipe was fixed by hand. Benchmarking ECG FMs finds a 3.8M-parameter ECG-CPC model competitive with far larger models and reports 3.3x to 9x label efficiency. That makes cheap, many-trial fine-tuning searches realistic. Active Learning with Foundation Model Priors notes that its imbalance-related thresholds were tuned by hand, which is another sign that tuning under skew is currently done without guarantees.
Research Program
We first build the testbed. The backbone is a CPC-pretrained ECG encoder in the style of Mehari and Strodthoff or ECG-CPC. The downstream task is the PTB-XL 24-subclass and 71-statement tasks (PTB-XL), run on the official patient-stratified folds at label fractions of 1%, 5% and 10%. At these fractions several subclasses have single-digit validation positives.
The search space is genuinely tree-structured. At the top level the adaptation mode is linear probe, partial unfreezing, or full fine-tuning. Conditional on that choice, the space includes unfreeze depth and layer-wise learning-rate decay. The imbalance handling is chosen among class-weighted loss, focal loss, or minority oversampling, each with its own conditional parameter. Learning rate, weight decay and epochs are shared across branches.
We then run a controlled overfitting study. We collect about 2,000 random configurations per setting and score each on the small validation set and on the large held-out test fold. This gives an offline lookup table, so any optimizer can be replayed at negligible cost against the same trial results. From it we measure two quantities as functions of trial count and minority validation count. The first is the optimism gap: the chosen configuration's validation score minus its test score. The second is selection regret: the best test score among all configurations the optimizer visited, minus the test score of the one it chose.
Finally we implement rank-uncertain TPE and its constrained variant on top of the Optuna-style TPE. We compare these against the baselines below on the same lookup tables. We then confirm with live runs on the best-performing arms.
Baselines
The naive reference point is no search. We fine-tune with the fixed two-stage recipe of Mehari and Strodthoff and a class-weighted loss, using the same encoder, labels and folds. Any tuner that fails to beat this default on test has done harm. Our central empirical claim is that standard TPE sometimes falls into that category at 1% labels.
The strongest existing method is standard TPE with a hard quantile split and argmax final selection. This is the procedure of SCL-TPE ( Supervised contrastive representation learning with tree-structured parzen estimator Bayesian optimization for imbalanced tabular data), extended to our conditional space. It gets the same trial budget (100 and 300 trials), the same validation set and the same $\gamma$. For the constrained task it is paired with c-TPE using hard feasibility labels. No reported result was found on Lacuna for TPE-tuned fine-tuning of PTB-XL at low label fractions. The published anchors on PTB-XL are 0.9418 macro AUC for CPC fine-tuning and 0.9318 for supervised training (Self-supervised representation learning from 12-lead ECG data Self-supervised representation learning from 12-lead ECG data), and about 0.939 for a CNN ensemble ( Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL). All of these are at full labels, so they bound what is achievable rather than serving as low-label targets. For a tabular sanity check we rerun SCL-TPE's temperature search on Glass0 and Lymphography. There the reported result is 0.837 accuracy and 0.842 AUC on Glass0 for SCL-TPE (from the same paper). We check how much of that lead survives test-set replication across many splits.
We also include random search at the same budget, since noise may erase any benefit of model-based search. Repeated stratified cross-validation with standard TPE is matched on total fine-tuning compute, which means fewer trials.
Three ablations isolate the key ingredient. The first is standard TPE with only lower-confidence-bound final selection. The second is soft weighting from independent, unpaired per-trial bootstraps. The third is a hard split on bootstrap-mean scores.
Rank-uncertain TPE counts as beating the baselines only if it meets two conditions at 1% and 5% labels. It must reduce selection regret relative to standard TPE, with a paired test across at least 20 fold-by-seed replicates. And it must improve test rare-class macro AUC over both standard TPE and the no-search default. For the constrained variant it must lower the rate of test-set violations of the 95% specificity floor, without lowering sensitivity at that floor.
Why not just…?
Why not just use random search, since noise makes model-based search pointless anyway? This is a real possibility, and our lookup-table replay tests it directly. If standard TPE's selection regret equals random search's, noise has indeed flattened the landscape. Our method's value would then have to come from its final-selection rule. The first ablation separates exactly that. We expect TPE to retain an edge at 10% labels and lose it at 1%. Mapping where that crossover falls is itself a useful result.
Why not just replace each trial's score with its bootstrap mean? When all trials share one validation set, resampling shifts every score in a correlated way. The mean score therefore preserves almost the entire ranking, and a hard split on it barely changes which trials count as good. The third ablation tests this. If it matched our method, the paired soft split would add nothing and we would report that.
Why not just use repeated cross-validation? Averaging over several splits genuinely reduces noise, and at 10% labels it may win. It costs k times the fine-tuning compute, though, and it still leaves few positives per fold at 1% labels. We compare at matched compute. The prediction is that rank-uncertain TPE wins when the budget is tight and rare positives are fewest. If cross-validation wins everywhere, our recommendation becomes to simply spend compute on cross-validation.
Why not just skip tuning and use the default recipe? For the smallest label fractions this may truly be the best choice. The no-search baseline is included precisely so the study can say so if it is.
Decisive Checks
The first check is whether standard TPE's optimism gap grows with trial count and falls as minority validation positives increase. If it does not, the premise fails and the method is unnecessary. The second is whether rank-uncertain TPE's regret reduction exceeds that of the lower-confidence-bound-only ablation, showing that uncertainty helps the search and not just the final pick. The third is whether results transfer beyond ECG, to the long-tailed PathMNIST setting of PriorAL and to the KEEL tables. On those tables we expect part of SCL-TPE's reported margin to shrink under replication with fresh splits.
References
- Data-Driven Bayesian Optimization Framework for Rapidly Developing Novel Wideband, Low-Profile Dipole Antenna With 3-D-Printed Technology
- Needle in a Haystack: Label-Efficient Evaluation under Extreme Class Imbalance
- Supervised contrastive representation learning with tree-structured parzen estimator Bayesian optimization for imbalanced tabular data
- c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
- Multiobjective Tree-Structured Parzen Estimator
- Self-supervised representation learning from 12-lead ECG data
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance
- PTB-XL, a large publicly available electrocardiography dataset
- Supervised contrastive representation learning with tree-structured parzen estimator Bayesian optimization for imbalanced tabular data
- Self-supervised representation learning from 12-lead ECG data
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL