Do Trade-off Curves Survive a Move to a New Hospital?

A clinical model rarely serves one goal. It has to catch rare events without flooding staff with false alarms, work for small patient subgroups as well as the majority, and handle rare diagnoses alongside common ones. Multi-objective methods describe these tensions with a Pareto front, the set of models where no goal can improve without another getting worse, and then let a hospital pick a point on that curve. The reproducibility literature, meanwhile, shows that clinical models built at one site often fail at the next. The widely deployed Epic Sepsis Model was marketed at an AUC of 0.73–0.83 and measured at 0.60–0.64 at external hospitals (All models are local). Nobody has asked the multi-objective version of that question. When a trade-off curve learned at one site is carried to another, does the curve keep its shape? Does the operating point a committee chose still sit on the curve, or is it now beaten by points it used to beat?

This note proposes measuring that directly on public critical-care data and splitting any loss into two parts. The first is a calibration part, which local threshold or output recalibration can repair cheaply. The second is a shape part, which requires changing the model's weights. The central claim is that front reproducibility should be reported as a separate quantity from per-objective accuracy drops. Whether a site needs to relearn its trade-offs or only re-threshold is an empirical question that can be settled, and this work aims to settle it on two concrete clinical tasks.

Core Claim

The proposal tests one hypothesis that could fail. Under a documentation-system and era shift, the hypothesis says a learned Pareto front degrades unevenly. The parts of the front that favor the rare or minority objective lose the most. Some dominance relations between operating points also reverse. Per-output recalibration on a small local sample does not remove this "shape" loss, but cheap local relearning of the front from that same sample does.

If recalibration alone recovers the target-site front, the conclusion is just as useful and simpler: sites need recalibration protocols, not new multi-objective machinery. The experiment is designed so that either outcome is a clean result.

Prior Work and Positioning

Three lines of work set up the question without answering it.

The first supplies the method. Learning the Pareto Front with Hypernetworks trains one network that takes a preference vector (how much to weight each goal) and outputs the weights of a full model. This yields a whole front in one run, reportedly 10x to 50x faster than training separate models. Its exact-Pareto variant (PHN-EPO) reaches the dented, non-convex parts of the front that weighted sums miss. Basaklar et al. give a lighter preference-conditioned network. Both are evaluated only on static, centralized benchmarks. We reuse them as front learners and add the cross-site test they never ran. The hypernetwork's memory cost grows with target-model size, but this is not a barrier here because the clinical models are small recurrent networks.

The second line motivates the question. All models are local argues that clinical data reflects local charting templates and workflows. It proposes "silent mode" shadow deployment followed by local fine-tuning, but treats adaptation as single-objective and notes that no accepted measure exists for when a new site is "similar enough." Learning with Impartiality maps accuracy–privacy–fairness surfaces on CheXpert and face datasets. It shows that tuning hides real trade-offs, but it never checks whether those surfaces transfer. Our contribution is the transfer test plus a front-level adaptation step built on the shadow-mode sample that "All models are local" already recommends collecting.

The third line constrains how we evaluate. Reproducibility and Baseline Reporting for Dynamic Multi-objective Benchmark Problems shows that dynamic multi-objective papers beat a "random restart" baseline that is actually the weakest option, while doing nothing often does better. The lesson for us is not to compare against retraining from scratch on tiny local data and call that progress. Automated Medical Coding on MIMIC-III and MIMIC-IV shows that a wrong macro-F1 formula, unstratified splits and untuned 0.5 thresholds produced fake architectural gains. It also shows that a tuned Bi-GRU beats fancier models. We inherit three rules from it: its corrected metric, its stratified MIMIC-III clean split, and mandatory threshold tuning for every method. MIMIC-Extract supplies standardized preprocessing and leakage-safe "gap time" labels. Its finding that Random Forests rival LSTMs is a further warning against assuming complexity is needed.

Research Program

We build two testbeds where real conflict and a real site-like shift both exist in public data.

Testbed 1: ICU interventions. This uses MIMIC-Extract time series. The objectives are early-warning quality for ventilation onset, early-warning quality for vasopressor onset, and worst-subgroup loss across demographic groups. The source is MIMIC-III's earlier documentation-system era and the target is its later era. Because the documentation system changed between these eras, this shift mimics the template differences blamed for the sepsis model failure. A second target is MIMIC-IV, which covers later years.

Testbed 2: medical coding. Source is MIMIC-III clean and target is MIMIC-IV, with ICD codes recoded to match. The two objectives are loss on frequent codes and loss on rare codes. The coding replicability study identifies label sparsity on rare codes as the true bottleneck, so this conflict is real rather than invented.

Before anything else, we check that each pair of objectives actually conflicts. Separately trained single-objective models must disagree in the target metrics, or the front collapses toward a point and that objective pair is dropped.

On each source we learn a front with PHN-EPO, the preference-conditioned network, and a grid of separately trained weighted-sum models. We evaluate each on a fixed, pre-registered set of preference rays. Then we compute three things at the target. The first is the transported front: the source models' points evaluated on target data. The second is the recalibrated front: the same models after per-output isotonic recalibration and threshold retuning on a small labeled target sample of size $n$. The third is the oracle front: models trained on the full target data.

The quantities we report are listed here. The normalized hypervolume gap between oracle and transported fronts is the share of the target's achievable trade-off region that the transported front fails to cover. We report it with one fixed reference point declared in advance and bootstrap intervals over patients and seeds. The split of that gap into a calibration share (closed by recalibration) and a shape share (remaining) is the decomposition at the heart of the proposal. The ray-wise gap profile shows which preferences lose most. The dominance-flip rate is the fraction of operating-point pairs whose ordering reverses between sites. We also test whether simple shift measures between sites predict the shape share, as one step toward the population-similarity metric that "All models are local" calls for.

The adaptation method is front-level local relearning. We fine-tune only the preference-conditioning part of the source front learner on the $n$-patient shadow sample, sampling preferences from a Dirichlet distribution as in the original hypernetwork work. This adapts all operating points at once without new training per point. We sweep $n$ from a few hundred patients up to several thousand to locate where each method becomes worthwhile.

Baselines

The naive reference point is to do nothing: deploy the source front unchanged. Following the dynamic-benchmark audit, this counts as a serious baseline, not a strawman. Any method that cannot beat it at small $n$ gives no reason to adapt at all.

The strongest cheap competitor is local recalibration. It applies per-output isotonic recalibration and per-objective threshold retuning on the same $n$ target patients, applied to every source operating point. This is the fix "All models are local" implicitly recommends, and the coding replicability study shows that threshold tuning alone changes conclusions.

The strongest weight-level competitor is single-point local fine-tuning. Here a tuned source Bi-GRU or GRU is fine-tuned at the one preference the site deploys, using the same $n$ patients. We also include an equal-budget grid of fine-tuned weighted-sum models. For the source models, the only reported discrimination figures on Lacuna are from MIMIC-Extract: intervention macro AUROC between 0.90 and 0.94 for Random Forests and LSTMs, with AUPRC often below 0.50, and mortality AUROC between 0.85 and 0.90. We will reproduce these before any transfer experiment. No reported hypervolume result was found on Lacuna for Pareto hypernetworks on any benchmark, and none for MIMIC-IV transfer, so all front-level numbers here are new.

Full local retraining of PHN-EPO on the $n$ patients is run as the "restart" comparison, and is expected to be weak at small $n$. The oracle front trained on all target data is an upper bound, not a competitor.

Two ablations isolate the key ingredient. In the first, the relearning step is trained with linear weighted sums instead of the exact-Pareto solver; this tests whether reaching non-convex regions matters after transfer. In the second, preference sampling is replaced by a single fixed preference; this tests whether adapting the whole front helps the deployed point.

Fairness is enforced in four ways. All methods share the same splits, labels, number of target labels $n$, and hyperparameter-search budget measured in GPU-hours. All use the corrected macro F1. All are tuned on a target validation set disjoint from evaluation.

Front-level relearning "wins" only under a strict condition. It must close a significantly larger share of the oracle hypervolume gap than local recalibration at the same $n$. It must also match single-point fine-tuning at the deployed preference within confidence intervals.

Why not just…?

Why not just recalibrate thresholds at the new site? This is the real threat, and it is the main experiment rather than a side check. If the shape share of the gap is near zero on both testbeds, the front moved without changing its shape, and the correct recommendation is a recalibration protocol. We would report that plainly. The reason to expect a nonzero shape share is that the coding study traces errors to rare-label sparsity in learned representations, not to thresholds. Representation errors cannot be undone by remapping outputs one at a time. Only the comparison at matched $n$ decides.

Why not just fine-tune the one model the hospital deploys? Hospitals do deploy one point, and if single-point fine-tuning matches front relearning at that point, it is the simpler choice for that use. The front matters in two cases. The first is when the site's preferred trade-off itself changes, for example when alarm-fatigue tolerance tightens. The second is when one needs to know whether the deployed point is still efficient, which is exactly what the dominance-flip rate measures. We report the deployed-point comparison separately so this alternative can win where it should.

Why not just report each objective's metric drop separately? Per-objective drops cannot detect dominance reversals or uneven losses along the curve. Two fronts can show identical average drops while one loses its entire high-sensitivity end. The ray-wise profile and flip rate test whether this happens. If they add nothing beyond per-objective drops, that is itself evidence the extra reporting is unnecessary.

Why not just sweep weighted sums instead of using a hypernetwork? Weighted sums cannot reach dented regions of the front, as shown in Cheng et al. and the hypernetwork paper. Still, with small models a grid is affordable. That is why the grid appears as an equal-budget baseline and the linear-sum ablation runs inside our own method.

Decisive Checks

The work succeeds if four things hold together on at least one testbed. First, the transported front shows a hypervolume gap with nonzero dominance flips. Second, a significant shape share remains after recalibration. Third, front-level relearning closes that remaining share at a moderate $n$. Fourth, the effect concentrates on the rare-code or minority objective.

The work fails cleanly if either of two things happens. Recalibration might close nearly all of the gap, in which case fronts are reproducible up to calibration. Or single-point fine-tuning might dominate front relearning at every $n$, in which case local adaptation does not need the multi-objective machinery. Either result changes what a hospital should demand when a vendor ships a "tunable trade-off" model.