Re-tune Only When the Data Has Changed Enough: Change-Point Detection Along a Dataset's Version History

Real datasets are never finished. Labels get corrected, extra annotators are added, and outliers are removed or imputed. Each such step creates a new version of the data. Teams then face a choice at every version. They can reuse the model and hyperparameters picked earlier, which is cheap but may be stale. Or they can re-run model selection and hyperparameter search from scratch, which is correct but expensive. The data-centric benchmark of Is one annotation enough? shows how costly this is. Enriching annotations raised average accuracy from roughly 55% to 72% across ten datasets. Even so, the authors kept hyperparameters fixed across every data version, because 5,500 GPU hours was already too much. Nobody knows whether those fixed settings were leaving accuracy on the table, or at which versions.

This note proposes to treat the version history itself as a time axis and to run joinpoint regression along it. Joinpoint regression is the epidemiologists' tool for fitting a trend as straight-line pieces joined at a few "knots" where the slope changes. At each new version we compute a handful of cheap signals: transferability scores for each candidate model, and short, low-cost training runs of the current best configurations, which act as sentinels. We then ask whether the latest version starts a new trend segment. Full re-tuning is triggered only when it does. The x-axis is not the version count. It is the cumulative amount of data changed, as recorded by the provenance log. A thousand relabeled images therefore move the axis far more than one typo fix. The claim is that the tuning landscape shifts in a few discrete jumps rather than continuously. If that is true, a knot detector on cheap signals recovers nearly the quality of re-tuning at every version for a small fraction of the compute.

Core Claim

Model rankings and good hyperparameters change piecewise along a dataset's provenance history. A segmented-regression test on cheap per-version signals, indexed by provenance-measured data change, identifies the versions where re-tuning pays off. Concretely, this policy should come within seed noise of per-version full hyperparameter optimization (HPO) at a small fraction of its cost. It should also beat three simpler policies at equal total compute: fixed hyperparameters, re-tuning on a fixed schedule, and warm-started re-tuning at every version.

The idea departs from each of the directions it draws on. Provenance systems record change as passive ledgers. Decibel versions relational tables inside the storage engine, with commit metadata overhead under 1%. openclean logs cleaning operations through histore so they can be rolled back. EXACT links training checkpoints to annotation versions. None of them feeds these records into a tuning decision; we use them as the measurement axis.

The closest use of piecewise-linear fitting in machine learning is Predicting the Emergence of Induction Heads. It locates the training step at which a capability abruptly forms, and it does so offline, over training steps within one run. We instead apply knot detection online, across data versions, to schedule compute.

Multi-fidelity HPO such as BOHB treats the dataset as a fixed snapshot. We keep BOHB as the inner search routine and only decide when to call it.

Research Program

Version chains. We represent a dataset's history as a chain of versions $D_0, D_1, \dots$. Branching histories of the kind Decibel stores are handled one root-to-leaf path at a time; merges are out of scope.

Provenance-measured distance. Each version gets a position $x_v$, the cumulative provenance distance. It is the fraction of labels or rows touched since $D_0$, read directly from the version log's diffs.

Cheap per-version signals. At each version we compute three things:

  • LogME and TransRate scores for every candidate backbone. These score frozen features against the labels and run about 3,000 times faster than fine-tuning, as reported for vision.
  • Short-budget training runs of the current top few configurations, the sentinels.
  • The rank agreement between these quantities and their values at the last tuned version.

Knot test. For each signal $s$ we fit the continuous hinge model

$$s(x) = \beta_0 + \beta_1 x + \beta_2 (x - \tau)_+ ,$$

where $(x-\tau)_+$ equals $x-\tau$ when $x>\tau$ and zero otherwise. The knot $\tau$ is restricted to the observed version positions. A knot is accepted at the newest version by a permutation test or the Bayesian Information Criterion (BIC), following the standard joinpoint procedure. We also accept a significant residual at the newest point, which catches a jump that has only one point after it.

Re-tuning. When a knot is accepted, we re-run BOHB on the new version. Its search is warm-started from the history of the segment just closed, and a new segment then begins.

Honest limits. Version chains are short, often five to fifteen points, and permutation tests have little power on so few points. We will therefore report how often the detector fires late, and compare against a simple residual threshold, before claiming the segment structure matters. We will also test the hinge model's assumption that change arrives as slope breaks rather than smooth drift. The test is to check whether the knots it finds line up with versions where re-tuning actually helped.

Baselines

Fixed hyperparameters. This is the do-nothing reference, and it is exactly what the DCIC study did (Is one annotation enough?). One BOHB search is run on $D_0$, and its best model and configuration are reused unchanged at every later version. Its reported reference point is average accuracy rising from about 55% to 72% as annotations are added, under fixed hyperparameters. Any policy that spends extra compute must beat this per-version accuracy by more than seed variance to justify its cost.

Per-version full BOHB. BOHB is run from scratch at every version with the same budget bounds. This is the quality ceiling, and it is what our ground truth is built from. BOHB's reported figures show why it is the right inner optimizer: it reached equivalent performance up to 100 times faster than Hyperband on the poker task, and reached 2.78% CIFAR-10 test error in 33 GPU days (BOHB). We count beating this baseline as matching its mean accuracy within seed noise while using at most a third of its GPU hours.

Warm-started BOHB at every version. This is the strongest cheap competitor. At each version it re-runs BOHB with a reduced budget, seeded from the previous version's evaluations. It is given exactly the total compute our policy ends up spending, spread evenly across versions. No reported result was found on Lacuna for warm-started multi-fidelity HPO across dataset versions. Beating it means higher mean accuracy at equal total compute. The difference should be concentrated at versions our detector flags.

Two ablations isolate the key ingredient. The first is periodic re-tuning every $k$ versions, with $k$ chosen so the number of re-tunes matches our policy's. This tests whether the timing of re-tuning matters or only the count. The second replaces the joinpoint test with a plain threshold on the sentinels' accuracy drop. This tests whether segment structure adds anything beyond a monitor. A third ablation indexes by version count instead of provenance distance, testing whether the provenance axis earns its place.

Why not just…?

Why not just warm-start the search at every version? This is the most serious threat. If the tuning surface drifts smoothly, spreading budget evenly with warm starts could match or beat a policy that waits for a knot. The equal-compute comparison settles it directly. We expect our policy to win only if gains from re-tuning are concentrated at a few versions. The ground-truth analysis measures that concentration independently: for each version, how much the re-tuned configuration beats the reused one. If the gains turn out to be spread evenly, we will report that warm-starting everywhere is the better policy.

Why not just keep hyperparameters fixed, since cleaner data usually helps anyway? Wiemerslage et al. show that architecture comparisons flip once models are properly tuned. In their study, LSTMs matched or beat Transformers on character-level tasks only under thorough search. A fixed configuration may therefore keep the wrong backbone as the data changes. Our ground truth measures how often the per-version best model differs from the one chosen at $D_0$. That frequency is the empirical case for or against doing anything at all.

Why not just re-tune whenever the sentinel runs lose accuracy? This may work as well as the joinpoint test, and the threshold ablation exists to find out. The joinpoint version can do two things a threshold cannot. It can catch a slope change before any single step crosses the threshold. It also indexes by provenance distance, which separates many small edits from one large one. If the threshold matches it on every setting, the contribution reduces to "cheap sentinels plus provenance-indexed monitoring," and we will say so.

Why not just rely on static transferability scores to re-rank models at each version? LogME and TransRate are cheap enough to compute at every version. However, they score frozen features under one fixed linear head, and they say nothing about the fine-tuning hyperparameters. This is the confound Wiemerslage et al. document. We will report the Kendall rank correlation between the static-score ranking at each version and the tuned ranking. We will also report whether the static scores alone fire at the right versions. They are included in the detector's inputs precisely so this can be measured.

Proposed Experiments

Annotation histories. The main setting uses the DCIC datasets, all under 15,000 images. Increasing annotation counts per image provide a natural version chain, and the fraction of changed soft labels gives the provenance distance. For cost, we restrict ground-truth per-version BOHB to four datasets, including CIFAR-10H and Benthic, with about six annotation levels each. The candidates are a small zoo of ImageNet-pretrained backbones, scored by LogME and TransRate and fine-tuned with BOHB.

Cleaning histories. The second setting uses OpenML tabular tasks drawn from HPOBench, with XGBoost, random forest and MLP models. Version chains are built by injecting documented corruptions and then applying openclean repair operators one at a time. Each repair is logged in histore, so the provenance distance comes from the real ledger. HPOBench's tabular surrogates assume fixed datasets, so real training is required here. These models are cheap enough to allow many chains and seeds.

Metrics. We measure mean and worst-case accuracy regret against per-version full BOHB, and total GPU or CPU hours. We also compute the detector's precision and recall against "shift versions." A shift version is one where re-tuning beats reuse by more than two seed standard deviations. Finally, we measure Kendall rank correlation of the chosen model rankings against the tuned ranking.

Decisive checks. Three results would make the case, and each could fail:

  1. Most of the per-version re-tuning gain falls on a small share of versions.
  2. Our policy captures that gain at a third or less of full-HPO compute.
  3. Our policy beats both even-budget warm-starting and the threshold monitor at equal compute.

If the first fails, the premise is wrong. If the third fails against the threshold monitor, joinpoint regression is unnecessary and the simpler monitor is the result.