Equivalent Feature Sets as Label-Free Drift Detectors and Repair Tools in Clinical Prognosis

Clinical prediction models decay after deployment. Lab assays get recalibrated, coding rules change, scanners are replaced, and a model that ranked patients well in 2012 ranks them worse in 2016. The usual fix is to refit the model on recent labeled data. For prognosis, though, the labels are survival outcomes that take years to arrive, so the model has to cope with drift long before anyone can retrain it. This proposal rests on a fact about clinical data that has been studied only in static settings: the same prognostic information is often recorded several times, in different variables and often in different modalities. A lab value, a diagnosis code and a chest X-ray finding can each carry the same signal about the outcome. Methods for equivalent feature set discovery find these interchangeable predictors, meaning distinct minimal sets of variables that predict equally well.

The core idea is that interchangeable predictors check each other. Suppose two variables were equivalent in training data because they measured the same underlying quantity. If the relationship between them changes in new, unlabeled deployment data, then at least one of them has been re-measured or re-coded. We can detect this without waiting for outcomes. We can then repair the model by switching to the equivalent set whose members still agree, rather than retraining or discarding signal. Put in two sentences: we use the redundancy among equivalent feature sets as an alarm that needs no labels, and we use the same redundancy to swap out a broken variable for an intact substitute. Multimodal data makes this practical, because a substitute for a drifting variable is often found in a different modality that drifts for different reasons.

Core Claim

Formally, let $Y$ be the outcome, $S$ a set of variables that every minimal predictive set contains, and $Q = \{x_1, \dots, x_m\}$ an equivalence queue. A queue is a group of variables any one of which can complete $S$ into a minimal predictive set, because each makes the others conditionally uninformative:

$$x_i \perp Y \mid \{x_j\} \cup S \quad \text{for all } i \neq j \in Q.$$

Checking this condition at deployment time would require $Y$. The equivalence, however, usually arises from a tight relationship among the queue members themselves, and that relationship can be observed without labels. For each member we fit, on historical data, a model $g_i$ predicting $x_i$ from the other members and $S$. At deployment time $t$ we track the coupling statistic

$$D_i(t) = \mathcal{L}_t\big(x_i, g_i(x_{-i}, S)\big) - \mathcal{L}_{\text{train}}\big(x_i, g_i(x_{-i}, S)\big),$$

where $\mathcal{L}_t$ is the held-out prediction loss on unlabeled data from window $t$. A member whose $D_i$ rises while the other members' couplings stay intact is flagged as technically drifted. The deployed predictor is then switched to an equivalent set that excludes it.

The claim to be tested has two parts. First, coupling breaks among equivalent features precede and predict the loss of a feature's conditional predictive value, which can only be measured later once labels arrive. Second, switching signatures on these alarms preserves discrimination under label delay better than pooled retraining with recalibration, invariance penalties, or simply dropping drifting features.

The method has a clear blind spot. It cannot see pure concept shift, where $P(Y \mid X)$ changes while the variables keep their mutual relationships. It targets the technical and administrative drift that Evaluating Model Performance in Medical Datasets Over Time found behind sharp performance crashes, such as the SEER extent-of-disease recoding.

Where This Sits

Three bodies of work supply the pieces, and none of them assembles them.

The equivalence literature supplies the method. Statistically Equivalent Signatures (SES) Feature Selection with the R Package MXM: Discovering Statistically Equivalent Feature Subsets builds equivalence queues with conditional independence tests, including censored-survival tests. On Vijver-2002 it found dozens of survival signatures whose performance varied with a median coefficient of variation below 5%. TMFBS Extending greedy feature selection algorithms to multiple solutions extends greedy forward-backward search to multiple solutions. It sorts variables into indispensable and replaceable ones, which corresponds to our $S$ and $Q$, and it found multiplicity in nearly 40% of the UCI datasets it tested. ChronoEpilogi ChronoEpilogi: Scalable Time Series Selection with Multiple Solutions carries equivalence into time series but explicitly assumes a stationary data-generating process and names non-stationarity as open. All three treat the set of equivalent signatures as fixed. We treat it as a quantity that changes over time and can be measured.

Two applied works motivate the substitution idea under static constraints. The COVID-19 study Automated machine learning optimizes and accelerates predictive modeling from COVID-19 high throughput datasets swaps hard-to-run assays for routine chemistry panels. The severe-weather work Yu et al. substitutes atmospheric precursors when sensors fail. In both, a human knows which input has failed. Our contribution is to detect the failure automatically from the data.

The temporal shift literature motivates the work and also constrains it. Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine found that recalibration restores calibration but not discrimination. It also found that the one feature-level method that preserved discrimination was manual expert grouping of unstable codes into stable clinical concepts, while automated PCA-style extraction failed. Our method is an automated version of that expert grouping. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine is the main constraint. On MIMIC-IV from 2008 to 2019, invariance methods (methods that train the model to ignore features whose behavior changes across time periods) never beat ordinary training on pooled history, because the changing features still carried signal. This rules out any design that penalizes drifting features. Ours removes a feature only when an intact equivalent substitute exists, so signal is preserved by construction. EMDOT Evaluating Model Performance in Medical Datasets Over Time supplies the evaluation protocol: train on data up to time $t$, test after $t$.

Multimodal prognosis supplies the setting. Pathomic Fusion Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis and SlotSPE Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis are validated only on static splits. SlotSPE handles a missing modality by reconstructing genomics from histology, but it cannot handle a modality that is present and silently corrupted. Continually Evolved Multimodal Foundation Models for Cancer Prognosis adapts to evolving streams through low-rank weight updates. That requires labeled new data, and it never asks which inputs are interchangeable. We do not propose a new fusion architecture. Our point is that the cross-modal redundancy these models exploit can also serve to detect drift.

Research Program

The first component estimates equivalence over time. We run SES with survival and binary conditional independence tests separately within each temporal epoch and on pooled history. We measure how queues persist, split and merge across epochs, using Jaccard overlap of queue memberships and of the Multiple Solutions Graph, TMFBS's compressed graph of all equivalent sets. This is the first direct measurement of whether equivalence structure is stable in real clinical data. If it is not, the program stops here, and that finding is itself reportable.

The second component builds cross-modal queues. On MIMIC-IV we combine structured labs and vitals, diagnosis codes, and MIMIC-CXR image embeddings. Conditional independence tests cannot run on raw image embeddings, so we reduce each embedding to a small number of components, around twenty, using unsupervised reduction fitted only on pre-$t$ data. We admit cross-modal queue members only when the conditional independence tests support them. Because SES is sensitive to its significance threshold and to the maximum size of the conditioning set, we report results across a grid of both. We present the linear-test and kernel-test versions as an explicit ablation, since practical scalable equivalence methods rely on linear models.

The third component is the monitor and the switching rule. For each queue we fit the coupling models $g_i$ on pre-deployment data and compute $D_i(t)$ on unlabeled deployment windows. A permutation threshold calibrated on pre-deployment windows controls false alarms. When a member is flagged, the predictor moves to the best-validated signature that excludes it. That signature was already fitted on historical data, so switching needs no new labels. For queues with only two members, we attribute the drift to whichever member's coupling with the indispensable set $S$ changed. If neither member's coupling changed, or both did, the alarm is reported without a switch.

Baselines

The naive reference is a frozen model: a single-signature Cox or logistic model, selected by LASSO on pre-$t$ data and never updated. It shows how much drift costs.

The strongest existing competitor is ordinary training on all pooled history, followed by recalibration on the most recent window whose labels have matured. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine found pooled training (their ERM[08–16]) to be the best strategy, beating every invariance and domain adaptation method. The file reports the size of the decay, a sepsis AUROC drop of up to 0.090, but no absolute AUROC for the pooled model. For SEER and OPTN, no reported result was found on Lacuna for an absolute temporal-backtest AUROC or C-index of any baseline. We will therefore reproduce this baseline in the EMDOT protocol, with the same feature pool, the same label delay and the same hyperparameter budget as our method. Beating it means a higher time-averaged C-index or AUROC across post-$t$ windows, with paired bootstrap confidence intervals that exclude zero, and no worse calibration slope after both methods are recalibrated.

We also run IRM, the invariance-penalty method benchmarked by Guo et al., to confirm their negative result in our setting. For the cancer static check, the reference is the published Pathomic Fusion C-index of 0.826 on TCGA-GBMLGG. That figure is not a temporal number, and we use it only to confirm that our reduced-feature pipeline is competitive.

Two ablations isolate the key ingredient. The first, drop-drifting-features, removes any feature whose marginal distribution shifts, which tests whether equivalence matters at all. The second, an ensemble of all signatures, averages predictions across every equivalent signature with no monitor, which tests whether the alarm adds value beyond redundancy alone.

Why not just…?

Why not just retrain on pooled data and recalibrate? For short-horizon outcomes this is a strong option, and we expect it to be hard to beat on MIMIC in-hospital mortality when labels arrive quickly. For long-horizon survival outcomes in SEER and OPTN, the most recent windows have no mature labels. In addition, Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine shows that recalibration does not restore lost discrimination. The label-delay sweep in Experiment 3 settles the question by showing the delay below which pooled retraining wins.

Why not just drop features that drift? The robustness paradox in Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine shows that drifting features often carry signal. Many drifts are also population changes that leave a feature's prognostic meaning intact. Our monitor distinguishes these cases, because a population shift moves the equivalent features together while a re-coding breaks their coupling. If the drop-drifting ablation matches our method, the equivalence machinery is unnecessary, and we will say so.

Why not just average over all equivalent signatures? This is the most serious threat. Averaging dilutes a single corrupted variable without needing any detection. However, averaging also keeps the corrupted variable in every signature that uses it, and when a queue has only two members, half the ensemble is affected. Experiment 4 injects corruptions with known severity and reports where the ensemble matches switching. If the two match on real backtests, we reduce the claim to the monitor's value as a diagnostic and recommend the ensemble for repair.

Proposed Experiments

Experiment 1 tests whether equivalence persists. We run SES per epoch on SEER Breast, Colon and Lung (1975 to 2013), OPTN liver transplant (2005 to 2017), and MIMIC-IV (2008 to 2019, in three-year windows). We report queue persistence and check whether the breaks line up with the known SEER extent-of-disease recoding.

Experiment 2 tests whether the alarm works without labels. Once labels mature, we compute in hindsight whether each queue member's equivalence held. We then score $D_i(t)$ as a detector of broken equivalence, reporting AUROC and lead time in windows.

Experiment 3 is the main repair comparison in EMDOT backtesting. We impose label delays of zero, one, three and five years, and compare all baselines on C-index or AUROC, calibration slope and Brier score for in-hospital mortality and sepsis in MIMIC, and for survival in SEER and OPTN.

Experiment 4 is a semi-synthetic check with known ground truth. In MIMIC-IV and MIMIC-CXR we inject unit changes, category recodings and scanner-like shifts into the image embeddings, each applied to one modality at a time.

Experiment 5 is a static feasibility check on TCGA-GBMLGG and TCGA-KIRC. It tests whether reduced histology features enter the same equivalence queues as genomic signatures, which is a precondition for cross-modal substitution in oncology. We do not make temporal claims on TCGA, because the file documents no temporal split for it.

The decisive outcomes are three, any of which would sink the full claim. Equivalence queues could prove unstable across epochs. The coupling alarm could fail to beat chance at detecting equivalence breaks. Or switching could fail to beat both pooled-plus-recalibration and the ensemble of all signatures under multi-year label delay.