Modern ML toolkits promise that you can swap a part, such as a feature extractor, a pre-training objective, or a loss head, without touching anything else. The software keeps that promise: the code still runs. What nobody measures is whether the hyperparameters of everything downstream still work. These are settings like the classifier head's learning rate, the margin in a face-recognition loss, or the temperature of a probe. If they stop working, every "plug-and-play" comparison carries a hidden cost. The new part either gets an unfair handicap, because it is run with settings tuned for the old part, or an unpriced advantage, because someone quietly retuned for it.
This note proposes to measure that cost directly and then to explain it. We will split the retuning a swap forces into two parts. One part comes from simple changes in feature scale, and fixing the scale at the interface should remove it. The other part comes from changes in the shape of the representation. We then test whether that second part can be predicted before any downstream labels are seen. The deliverable is a measurement protocol, a decomposition, and a cheap label-free test. The test tells a practitioner which downstream hyperparameters to retune after a swap and which can be safely kept.
Core Claim
Two strands of prior work motivate the claim. Towards Modular Machine Learning Pipelines shows that a downstream model silently adapts to the particular errors of its upstream model, so improving the upstream part can hurt the whole pipeline. It measures this only in loss, mostly on synthetic two-stage pipelines, and never asks what the downstream tuning burden becomes. A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning defines two quantities. The first is sensitivity, the penalty for using one fixed configuration everywhere instead of tuning per setting. The second is effective hyperparameter dimensionality, the number of hyperparameters that must be retuned to recover 95% of tuned performance. That work treats each algorithm as one monolithic block across environments.
We transpose both quantities from "across environments" to "across module swaps". Let $P(u, h)$ be the validation performance of a pipeline whose upstream module is $u$ and whose downstream hyperparameters are $h$. Let $h^*(u)$ be the best $h$ found for $u$ under a fixed search budget. For a swap from $u$ to $u'$ we define the swap penalty
$$\Phi_{\text{swap}}(u \to u') = \frac{P(u', h^*(u')) - P(u', h^*(u))}{P(u', h^*(u')) - P_{\text{ref}}},$$
where $P_{\text{ref}}$ is a fixed weak reference score used for normalisation. In words, $\Phi_{\text{swap}}$ is the fraction of the new module's tuned gain lost by keeping the old settings. We also define $d_{\text{swap}}$, the smallest number of coordinates of $h^*(u)$ that must be re-searched to recover 95% of $P(u', h^*(u'))$.
The claim has three parts:
- With raw interfaces, $\Phi_{\text{swap}}$ is well above seed noise for common swaps.
- Standardising feature scale at the interface removes a substantial but incomplete share of it.
- The remaining share is concentrated in a few downstream hyperparameters, and label-free statistics comparing old and new representations on the same inputs can identify those hyperparameters.
Research Program
Measurement grids. We build swap grids in three existing modular toolkits, chosen because each already separates an upstream part from a downstream part through a fixed interface.
- Face recognition. In FaceX-Zoo, which separates backbones (ResNet, MobileFaceNet, HRNet, Attention-56) from supervisory heads (ArcFace, CosFace, CircleLoss), the upstream module is the backbone. The downstream hyperparameters are the head's margin and scale, the learning rate, and weight decay.
- Time series. In UniTS, the upstream module is the self-supervised pre-training objective (contrastive at three granularities, masked prediction, or hybrid). The downstream module is a classification or anomaly head on HAR, Fault Detection, and Server Monitoring.
- Graphs. Using the evaluation suite of Overcoming Pitfalls in Graph Contrastive Learning Evaluation, the upstream module is the contrastive encoder (DGI, GRACE, Barlow Twins, SUGRL, SFA, COSTA). The downstream module is the evaluation probe and its regularisation.
Estimating $h^*$ without bias. Each upstream–downstream pair receives the same random search over the downstream space. We estimate $h^*$ on one set of seeds and score it on held-out seeds. This matters because selecting and scoring on the same runs inflates the apparent cost of keeping the old settings.
Robustness of the definitions. Following the caveat in the sensitivity paper that its dimensionality measure depends on grid design, we report $d_{\text{swap}}$ under two grid resolutions. We also report whether its value changes between them.
Decomposition. We repeat every grid under three interface conditions:
- Raw features.
- Features rescaled to fixed per-dimension mean and variance on the training set.
- Features whitened, which removes both scale and correlation.
The drop in $\Phi_{\text{swap}}$ from condition 1 to condition 2 is the scale-driven share. Whatever survives condition 3 is the geometry-driven share.
Label-free predictor. For every swap we compute, on unlabelled inputs, a small set of comparisons between old and new representations:
- similarity of the two representations (linear CKA, centred kernel alignment);
- overlap of each point's nearest neighbours under old versus new features;
- the change in effective rank;
- the change in angular spread of the features.
From these we train a small classifier to predict, per downstream hyperparameter, whether keeping its old value costs more than seed noise. We evaluate it on swaps held out by upstream module, not by random split. The predictor is then used as a policy: retune only the flagged hyperparameters with a reduced budget.
Baselines
Naive reference: keep everything. This is the "do nothing" practice the toolkits implicitly encourage. Downstream hyperparameters tuned for the original upstream module are reused unchanged after the swap. It is scored on the same held-out seeds, and any method must beat it in final performance to matter at all.
Upper reference: full retune. This is a fresh random search over all downstream hyperparameters for each new upstream module, with the same budget used to find $h^*$. It defines the ceiling. The target is to match it within seed noise at a fraction of its cost.
For scale, the absolute numbers FaceX-Zoo reports for a well-tuned pipeline are 99.88% LFW accuracy and 97.75% MegaFace Rank-1 with Attention-56. In the reduced-data regime we will actually use, it reports SST raising LFW from 91.77% to 99.38% on reduced MS-Celeb-1M. We use the reduced regime for compute reasons. Our full-retune scores there should land near the reported SST figure, which checks that our tuning is competent.
Strongest competitor for choosing what to retune: MOFA. MOFA samples configurations on an orthogonal Latin hypercube, a space-filling design that varies each hyperparameter independently of the others. It then uses marginal-variance importance analysis to freeze unimportant hyperparameters. It is the strongest competitor because it answers the same question, "which hyperparameters matter here", with real evaluations instead of label-free statistics. It reports a 12% improvement over random search on UCI Bayesian neural network tasks and up to 40% on Boston Housing at small budgets. Those numbers are on monolithic models, and no reported result on module-swap retuning was found on Lacuna.
We run MOFA's screening on the new pipeline with exactly the evaluation budget our predictor-guided policy spends. Our policy spends nothing on screening and everything on retuning, so the comparison is final performance at equal total evaluations. Beating MOFA means equal or better final score at equal budget across held-out swaps on at least two of the three testbeds.
Decoupling regulariser. The Causal Pipeline Optimization regulariser from Towards Modular Machine Learning Pipelines trains the downstream model so that its dependence on upstream errors is removed. We approximate its counterfactual terms as the paper suggests, with estimators of conditional average treatment effects (how much the outcome changes per input when the upstream output changes). If that regulariser already drives $\Phi_{\text{swap}}$ to noise, our predictor is unnecessary, and we report that.
Ablations isolating the key ingredient. Two ablations test whether the predictor adds anything:
- replacing its per-hyperparameter flags with "retune the same number of hyperparameters, chosen at random";
- replacing them with "always retune learning rate first".
Why not just…?
Why not just retune everything end-to-end after every swap? This is correct but expensive, and it scales with the number of swaps times the size of the downstream space. Exploiting Reuse in Pipeline-Aware Hyperparameter Tuning makes full retuning cheaper by caching shared upstream outputs. It reports up to 70x speedups with accuracy within 1% of plain random search. That caching is complementary, not competing: it reduces the cost per evaluation, while we reduce the number of hyperparameters needing evaluation. We will run both with caching enabled so that the comparison isolates the second effect.
Why not just normalise the interface so swaps stop mattering? This is the real threat, and the design treats it as a treatment arm rather than a strawman. The RL sensitivity study is mixed on this point. Zero-mean observation normalisation lowered sensitivity without costing performance. Advantage normalisation and percentile scaling raised peak return while making learning rates more sensitive. So normalisation helps only for some mechanisms. If whitened interfaces bring $\Phi_{\text{swap}}$ within seed noise on all three testbeds, the honest conclusion is that toolkits should standardise their interfaces, and the predictor is unneeded. We expect hyperparameters tied to the angular structure of features, such as the ArcFace margin and contrastive temperatures, to survive whitening. The experiment settles this directly.
Why not just use per-module transfer rules like Complete(d)P? Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration shows that per-module hyperparameters inside a decoder-only Transformer transfer across scale. Its gains came from structured scaling rules: 2.31x faster to target loss at 50M parameters and 1.32x at 7.2B. Those rules assume the same module type repeated with depth. A swap from a contrastive to a masked-prediction encoder is a change of kind, not of size, so no such rule applies. We note this rather than test it.
Decisive Checks
The first check is whether raw-interface $\Phi_{\text{swap}}$ exceeds the seed-to-seed spread on held-out seeds. If it does not, the hidden cost is negligible and the project stops.
The second check is the fraction of $\Phi_{\text{swap}}$ that survives whitening. Near zero means the normalisation answer wins. A substantial residual, concentrated in a small $d_{\text{swap}}$, supports the claim.
The third check is whether the label-free predictor flags the right hyperparameters on swaps held out by upstream module. We measure this by area under the ROC curve against "needs retuning" labels derived from the full grid. We also test whether the resulting partial-retune policy matches full retuning and beats MOFA screening at equal budget.
A useful secondary check comes from the graph testbed. The "diva" encoders identified by the graph benchmark (SFA, COSTA), which do well only in a narrow band of settings, should show larger $\Phi_{\text{swap}}$ than the stable "workhorse" encoders (Barlow Twins, SUGRL). If the ordering does not hold, sensitivity to an encoder's own settings and sensitivity to swapping it are different things. That would itself be worth reporting.
References
- Towards Modular Machine Learning Pipelines
- A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
- FaceX-Zoo: A PyTorch Toolbox for Face Recognition
- UniTS: A Universal Time Series Analysis Framework Powered by Self-Supervised Representation Learning
- Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks
- MOFA: Modular Factorial Design for Hyperparameter Optimization
- Exploiting Reuse in Pipeline-Aware Hyperparameter Tuning.
- Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration