Which Hyperparameters Does a Module Swap Break?

Machine learning code is increasingly built from swappable parts: a preprocessing step, an encoder, a loss term and an optimizer can each be replaced without touching the rest. Toolkits such as DeepDIVA and FaceX-Zoo make this possible by fixing the inputs and outputs each part must accept. These guarantees cover shapes and method names. They do not cover tuning. When a developer swaps one part, some of the previously tuned hyperparameters (the learning rate, a loss weight, a regularizer) silently stop being good, and the usual response is to retune everything from scratch.

We propose to measure, for each swap, which hyperparameters it actually invalidates, and to predict that set cheaply before paying for a full retune. The prediction comes from a short probe: brief, low-budget training runs that vary one hyperparameter at a time around the old best configuration in the new pipeline. The goal is a retuning procedure that touches only the hyperparameters the swap broke. The evidence that this is needed comes from Hyperparameter Transfer Across Developer Adjustments. In that work, pruning the search to hyperparameters that mattered before a code change often failed, because hyperparameters that did not matter beforehand became critical afterwards.

Core Claim

Our claim has two parts. First, the set of hyperparameters a swap invalidates is usually small and is determined by the swap itself. Second, a probe run around the old optimum identifies that set more reliably than importance estimates carried over from the old pipeline, and at a cost of a few percent of a full retune.

To make "invalidated" precise, we adapt the sensitivity measures of A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning. That paper scores whole algorithms by the performance lost when one fixed configuration is used everywhere, and by the smallest set of hyperparameters that must be tuned to reach 95% of tuned performance. We move both measures from "across environments" to "across a swap." Let $\theta^*$ be the best configuration of the old pipeline. Let $P'(\cdot)$ be the performance of the new pipeline. Let $P'^*$ be the performance of the new pipeline after full retuning. The swap's retuning dimension is

$$d_{\text{swap}} = \min \{ |S| : \max_{\theta_S} P'(\theta_S, \theta^*_{-S}) \ge 0.95\, P'^* \},$$

where $S$ is a subset of hyperparameters, $\theta_S$ are the values of that subset that we retune, and $\theta^*_{-S}$ holds every other hyperparameter at its old best value.

If $d_{\text{swap}}$ is typically small, the remaining question is whether we can find the right subset $S$ cheaply. That is what the probe is for.

Research Program

The first part of the program is a measurement study, which nobody in the file has done. We build swap families on pipelines whose module structure is explicit, and for each swap we compute $d_{\text{swap}}$ and the identity of $S$ by exhaustive tuning on cheap benchmarks.

On the tabular side, the pipelines follow the imputer, scaler, feature selector and model structure used in Exploiting Reuse in Pipeline-Aware Hyperparameter Tuning, on its 20 OpenML classification tasks and on tasks from AMLB. That earlier work treats the same kind of pipeline purely as a caching problem. It never asks how an upstream change moves the downstream optimum, which is the question we ask here. A swap in this setting replaces one stage's algorithm, for example a standard scaler with a quantile transform, or one feature selector with another.

On the deep reinforcement learning side, we reuse the PPO-on-Brax setup of Jordan et al. There, switching per-minibatch advantage normalization on or off is exactly a module swap, and the paper already reports that it raises sensitivity to other hyperparameters. We additionally use the eight developer-adjustment benchmarks of Stoll et al., which simulate code edits on fully connected networks, architecture search, XGBoost and SVM models.

The second part is the probe. For each hyperparameter, we run $k$ short, low-budget evaluations of the new pipeline. Each evaluation perturbs that one hyperparameter along a coarse grid while all others stay at $\theta^*$. We rank hyperparameters by the largest improvement observed and pass the top-ranked ones to a standard optimizer, which searches only them, warm-started from $\theta^*$.

The probe borrows its spirit from Taming Hyperparameter Sensitivity in Data Attribution, which replaced retraining sweeps with a cheap internal signal for choosing a single regularizer. It differs in two ways: it is generic rather than method-specific, and it targets selection of which hyperparameters to retune rather than their values.

The third part tests whether interface statistics explain the probe's findings. These statistics include the scale and spread of the tensor passed across the swapped boundary and gradient norms at that boundary. If they do, then a module could publish, alongside its input/output shapes, which downstream hyperparameters its output statistics act through. We treat this third part as exploratory. The proposal stands on the first two.

The proposal also constrains, and is constrained by, MOFA. MOFA freezes hyperparameters judged unimportant by factorial analysis within a single fixed pipeline. Our measurement directly tests when such freezing remains safe across a swap and when it does not, which is the tension between MOFA's results and those of Stoll et al.

Baselines

The naive reference is keep-old: deploy $\theta^*$ unchanged in the new pipeline. Its gap to $P'^*$ is the cost of a swap that nobody retunes, and it is the number our measurement study reports first.

The strongest existing competitors are the transfer strategies of Stoll et al., which were designed for exactly this setting. Best-first evaluates the old best configuration and then resumes a tree-structured Parzen estimator search. T2PE fits that estimator on old runs projected into the new search space. The paper reports that best-first and T2PE reach target performance 1.2x to 2.6x faster than starting from scratch on its adjustment benchmarks. To beat them, the probe-then-retune procedure must reach the same target (95% of $P'^*$) in less total compute, on the same benchmarks, with the same optimizer core.

We also run a from-scratch multi-fidelity optimizer (BOHB or DEHB as packaged in HPOBench). HPOBench reports that these methods lead black-box optimization under tight budgets, though it gives no single scalar we can quote as a target. We include MOFA as a second from-scratch reference; it reports +12% over random search on Bayesian neural network tasks.

Every method receives the same total budget, measured in training epochs or environment steps. For our method, the probe's own cost is charged against that budget. All methods use the same evaluation seeds and the same search ranges.

The key ablation replaces the probe's ranking with fANOVA importance from the old pipeline. This is Stoll et al.'s drop-unimportant strategy, and it isolates whether measuring in the new pipeline is what matters. A second ablation, described in the next section, is the one that could sink the method.

Why not just…?

Why not just warm-start from the old best and run T2PE? This is the most direct competitor, and it may be good enough when swaps are mild. However, T2PE still searches every dimension. If $d_{\text{swap}}$ is usually one or two out of eight or more hyperparameters, a search restricted to the right subset should converge much faster. The head-to-head time-to-target comparison settles this. If T2PE matches us, we will report that and conclude that subset selection adds nothing on top of a good transfer model.

Why not just retune everything with a multi-fidelity optimizer? HPOBench notes that plain black-box search catches up given enough budget, so with a large budget, full retuning is simply correct. Our claim concerns the regime where retuning happens after every edit, at a small fraction of the original tuning budget. We report curves of performance against budget, so the crossover point, if any, is visible.

Why not just run fANOVA on a handful of cheap random configurations of the new pipeline? This is the real threat. It also measures importance in the new pipeline, and it needs no special probe design. The difference is that random samples estimate global importance over the whole space, whereas our probe estimates how much changing one hyperparameter helps at the old optimum, which is the quantity relevant to repair. These can disagree. For example, a learning rate matters globally but may already be fine at $\theta^*$. We will give both methods identical low-budget allocations. If random-sample fANOVA ranks hyperparameters as well as the probe, we will drop the probe as a contribution, and the paper will rest on the measurement study plus the simpler fANOVA-based procedure.

Proposed Experiments

The first check is descriptive. Across all swap families, we report the distribution of $d_{\text{swap}}$, the keep-old performance gap, and how often the invalidated set $S$ contains hyperparameters that old-pipeline fANOVA ranked as unimportant. A high rate of such cases reproduces and quantifies the failure of drop-unimportant. On Brax, the advantage-normalization swap should produce a larger $d_{\text{swap}}$ than the observation-normalization swap, consistent with Jordan et al.'s finding that the former raises sensitivity and the latter slightly lowers it. If it does not, our swap-based measure disagrees with theirs, and that disagreement needs explaining.

The second check is predictive. We measure how often the probe's top-$m$ hyperparameters (for $m$ equal to the true $d_{\text{swap}}$) recover the true set $S$, compared with old fANOVA and with new-pipeline random-sample fANOVA.

The third check is end-to-end. We compare time to reach 95% of $P'^*$, and performance at fixed budgets of 5%, 10% and 25% of the original tuning cost, against keep-old, best-first, T2PE, BOHB/DEHB and MOFA. We report median and interquartile range over tasks and seeds.

The proposal succeeds if three things hold. First, $d_{\text{swap}}$ is usually much smaller than the total number of hyperparameters. Second, the probe recovers $S$ better than both fANOVA variants. Third, probe-then-retune beats T2PE's 1.2x to 2.6x speedups at small budgets. It fails cleanly if the invalidated set is typically large, since that would mean swaps genuinely require full retuning.