Deployed models see a stream of new, unlabeled inputs, and the true labels for those inputs usually arrive later. A diagnosis is confirmed weeks after admission, and a news category is assigned after editorial review. One way to adapt in the meantime is pseudo-labeling, where the model trains on its own predictions for the fresh inputs. The trouble is that a model trained on its own mistakes grows more confident in them, and under drift over time its confidence scores are the first thing to go wrong. A recent study of this delayed-label setting found that, under a fixed compute budget, pseudo-labeling and test-time adaptation fail to beat a naive learner that simply waits for the delayed labels (Label Delay in Online Continual Learning).
This proposal treats the late labels as something more than delayed training data. When the label for an input from $d$ steps ago finally arrives, we can check it against what the current model predicts for that same input. Accumulated over time, with old evidence discounted, these checks give a running estimate of how the model's pseudo-labels are currently wrong: which true classes get confused with which predicted classes, and at what confidence. We then use that estimate to correct the loss on the fresh, unlabeled inputs. In two sentences: delayed ground truth measures the model's current error pattern, and that measured error pattern makes self-training on today's data safe. Because the correction relies on a measured confusion pattern rather than on the model being usually right, it can still help in regimes where the model is wrong more often than not. Existing guarantees for drift-robust self-training break down exactly there.
Core Claim
Under temporal drift with label delay, a confusion estimate that is (i) computed from the current model's predictions on recently labeled inputs and (ii) weighted toward recent time steps can be used to correct the self-training loss on unlabeled inputs. Under a matched compute budget, this beats training on delayed labels alone, retrieval-based rehearsal, and confidence-thresholded self-training. The gains should be largest in two regimes. The first is when the model's mistakes are systematic and asymmetric across classes, which a single confidence threshold cannot represent. The second is when the accuracy of confident predictions falls below 50%.
The part that would be new is the closed loop. Delayed labels feed a time-weighted, class-conditional noise model, and that noise model directly reshapes gradient updates on the newest data. No paper in our search does this.
How This Sits Against Prior Work
Motivating evidence: the delayed-label setting. The closest work is Label Delay in Online Continual Learning. It formalizes streams in which only $y_{t-d}$ is revealed at step $t$. It shows that standard self-supervised learning, pseudo-labeling, and test-time adaptation lose to the naive delayed-label learner under compute constraints. Its fix, Importance Weighted Memory Sampling (IWMS), uses predictions on $x_t$ only as retrieval keys: it pulls similar old labeled examples from a replay buffer and rehearses them. IWMS therefore never trains on $x_t$ itself, which is the opposite choice from ours. The authors report only marginal gains on Yearbook, suggesting retrieval helps less when old labeled data no longer resembles the present. We keep $x_t$ in the loss and use the delayed labels to make that safe.
Motivating evidence: why confidence-based filters fail here. Three continual adaptation methods filter or stabilize pseudo-labels using the model's own confidence:
- CoTTA averages a slowly updated "teacher" copy of the model over up to 32 augmented views, and randomly resets 1% of weights to their original values.
- PLF adapts per-class confidence thresholds over time and regularizes the predicted class proportions.
- AnCon smooths each pseudo-label with an ensemble of past predictions and thresholds relative to a moving average of confidence. It proves error decreases if confident predictions are on average more than 50% accurate.
None of the three uses any external ground truth during adaptation. All three were tested on synthetic image corruptions rather than multi-year drift. AnCon's guarantee, by its own condition, says nothing once confident accuracy falls below one half.
The clinical literature explains why self-referential confidence is fragile under drift. A systematic review found calibration deteriorated in 11 of 15 clinical studies of temporal shift, and that recalibration restores calibration but often not discrimination (Guo et al., systematic review). Our design responds to both halves of that finding. We use outside labels to fix the confidence-to-correctness mapping, and we still update the representation, so the model can recover discrimination.
Constraining evidence: do not force invariance. On MIMIC-IV, domain generalization and unsupervised domain adaptation methods failed to beat plain empirical risk minimization (ERM, ordinary training on pooled data) across 2008–2019. A plausible reason is that time-varying features carry real signal (Guo et al., DG/UDA evaluation). Wild-Time similarly reports ERM as hard to beat across five temporal datasets. We therefore add no invariance penalty. The correction acts only on the target labels, not on which features the model may use.
Supplying the method. Time-weighted calibration comes from Non-Exchangeable Conformal Risk Control. That work discounts past calibration errors exponentially so that risk bounds survive drift, but it uses them only for monitoring and never feeds them back into training. The idea of letting a reliability estimate scale the guidance loss comes from AdaConG. AdaConG down-weights guidance signals whose conformal prediction sets are large, but it requires a fixed calibration split whose examples are interchangeable with test data. A temporal stream violates that assumption. We take the discounting from the first paper and the loss modulation from the second, and connect them through a refreshed, class-conditional estimate.
Research Program
The noise estimator. At step $t$, the learner holds a short buffer containing only the last $d$ steps of unlabeled inputs; it is not a full history replay. As each batch's labels arrive, the current model $f_t$ predicts on those now-labeled inputs. This forward pass is shared with the supervised update the naive learner already performs. We bin each prediction by its confidence $c$ and update a discounted count of (true class $j$, predicted class $k$) pairs within each bin. The estimated transition matrix is
$$\hat T_t^{(b)}[j,k] = \frac{\sum_{s \le t-d} \lambda^{\,t-d-s}\, n_s^{(b)}(j,k)}{\sum_{s \le t-d} \lambda^{\,t-d-s}\, \sum_{k'} n_s^{(b)}(j,k')}.$$
The symbols are defined as follows:
- $n_s^{(b)}(j,k)$ is the number of inputs from step $s$ with true class $j$, current-model prediction $k$, and confidence in bin $b$.
- $\lambda \in (0,1)$ is the discount factor that favors recent evidence.
- Each row of the matrix is the estimated chance that true class $j$ is pseudo-labeled as $k$.
To keep the estimate stable when counts are sparse, we shrink it toward a coarser confusion matrix that ignores confidence bins. The discount is chosen online by a recency-versus-variance rule in the spirit of the adaptive rolling window of Model Assessment and Selection under Temporal Distribution Shift. That paper picks a window length by balancing how fresh the data is against how noisy the estimate becomes.
The corrected loss. For an unlabeled input with pseudo-label $\hat y$ and confidence bin $b$, we use forward correction. The model's predicted class distribution is pushed through $\hat T_t^{(b)}$ before cross-entropy is taken against $\hat y$:
$$\mathcal{L}_t(x) = -\log\big[\,(\hat T_t^{(b)})^{\top} f_t(x)\,\big]_{\hat y}.$$
The model is thereby asked to explain its own noisy label through a measured noise process, rather than simply to agree with it. This matters below 50% accuracy. A binary model that systematically flips one class is wrong most of the time, yet the matrix describing its errors is still invertible and informative. A filter based on confidence or majority agreement, by contrast, would amplify the flip.
The open question we must answer. The estimate always lags the data by $d$ steps, so the measured noise describes the model on slightly old inputs. The program will measure how quickly the benefit decays as $d$ grows relative to the drift rate. We will also test a cheap check for when correction should switch off: when the matrix rows become near-uniform or poorly conditioned, the method falls back to the naive delayed-label update.
Tabular data. For tabular electronic health records, there is no augmentation scheme to lean on, and each admission appears once. The method needs neither augmentations nor repeated exposures, which is why it carries over to MIMIC-IV with an MLP or logistic-regression model. Logistic regression is included because EMDOT found simple models more robust to staleness than complex ones.
Baselines
All methods receive the same stream, the same label delay $d$, and the same backbone. They also receive the same per-step compute budget, counted in forward and backward passes, following the protocol of Label Delay in Online Continual Learning. Every baseline gets the same hyperparameter search size, tuned on the earliest time window only, so that no method is tuned on the future.
Naive delayed-label learner. This is the reference point and, by prior evidence, the strongest simple competitor. It spends its full budget training on $(x_{t-d}, y_{t-d})$ and ignores $x_t$. Prior work found pseudo-labeling and test-time adaptation could not beat it. On CLOC, a 100-step delay drops its accuracy from about 20% to about 11% (Botos et al.). Our method must exceed it on every dataset where delay causes a measurable gap. A gain that disappears once the naive learner gets the same compute would count as failure.
IWMS. This is the strongest existing method for this exact setting. On CGLM with a 10-step delay it is reported to recover 100% of the accuracy gap caused by the delay. On Yearbook it shows only marginal improvement, and no number for that is reported (Botos et al.). We use the published sampling procedure with a replay buffer of the size the paper specifies. Our method uses far less memory, and we report memory alongside accuracy. Beating IWMS means higher online accuracy on FMoW-Time and Yearbook at matched compute, and at least parity on CGLM. Since IWMS already closes the full gap on CGLM, the only aim there is not to lose ground.
Confidence-based self-training. We include AnCon (Joo & Klabjan), PLF (Tan et al.), and CoTTA (Wang et al.). Each is given access to delayed labels as extra supervised batches so that the comparison isolates how pseudo-labels are treated, not who sees more data. CoTTA's augmentation count is reduced until it fits the shared compute budget. On their home benchmarks, PLF reports 14.8% error on CIFAR-10-C against CoTTA's 16.2%. AnCon reports accuracy gains of 8% to 16% over vanilla self-training. No reported result was found on Lacuna for any of the three on Wild-Time or MIMIC-IV.
ERM on all history and on a sliding window. We retrain periodically on all delayed labels, and on the most recent window, to anchor against the findings of Guo et al. and Wild-Time. On the closest prior split, Wild-Time reports MIMIC-IV mortality AUC falling from 0.90 to 0.72.
Calibrated-threshold ablation. This ablation isolates our key ingredient. It uses the same delayed labels and the same discounting, but fits only a scalar recalibration of confidence (temperature scaling) and then keeps pseudo-labels above a calibrated-precision threshold. It has no class-conditional transition matrix and no loss correction. A second ablation keeps the full transition matrix but estimates it with predictions recorded at time $t-d$ instead of re-predicting with the current model. A third sets $\lambda = 1$ so that no recency weighting is applied. Our method is justified only if it beats the calibrated-threshold ablation.
Why not just…?
Why not just train on the delayed labels and ignore the unlabeled stream? This is the most serious threat, because Botos et al. found that every unsupervised use of $x_t$ they tried lost to it. Their failing methods, however, never used the delayed labels to judge pseudo-label quality. Our method spends almost no extra compute on that judgment, because the forward pass on labeled data is already being performed. The matched-compute comparison across a range of delays settles the question directly. If we cannot beat this learner at large $d$, where its staleness should hurt most, the proposal fails.
Why not just recalibrate confidence with the delayed labels and threshold? This is the simpler version of our idea and could plausibly match it. Calibration is exactly what drift breaks first, according to Guo et al.'s review. The difference is that a scalar recalibration assumes a confident wrong prediction is equally likely for every class. It discards inputs and cannot redirect them. A transition matrix can represent systematic confusions, such as one land-use class drifting toward another, and can still extract signal when most predictions are wrong. The calibrated-threshold ablation tests this. We expect the matrix to matter mainly on FMoW-Time with 62 classes and asymmetric drift, and on the sub-50% stress streams. We will report honestly if the scalar version suffices elsewhere.
Why not just use AnCon or PLF, which already handle continual shift? Both rely only on the model's own confidence. AnCon's error-reduction guarantee assumes confident predictions are more than 50% accurate. Under severe multi-year drift that assumption can fail; for example, source-only error on ImageNet-C is already 82.0% (Tan et al.). PLF's authors concede that its benefit fades over long sequences. The sub-50% stress experiments below are designed to show whether these methods diverge where ours does not. If they do not diverge, the argument for an external audit weakens to the large-delay regime alone.
Why not just use IWMS? IWMS works well when the replay buffer contains relevant old examples, as on CGLM. It helps little when the past no longer resembles the present, as on Yearbook, and it needs a large buffer. Our method should win precisely when nothing in the buffer resembles today's inputs. Reporting per-dataset results together with a measure of how close today's inputs are to the buffer (the cosine similarity IWMS itself uses) will show whether the two methods are complementary. If they are, combining them is a natural follow-up.
Proposed Experiments
Main comparison. We run the Wild-Time datasets in streaming mode (Wild-Time): Yearbook, FMoW-Time, HuffPost, arXiv, and MIMIC-IV mortality. We add CLOC and CGLM from Botos et al., plus MIMIC-IV sepsis using the four three-year windows of Guo et al.. That study reports a maximum sepsis AUROC drop of 0.090 for its baseline. Delays range from a few steps up to the 100-step setting where CLOC collapses.
Metrics. For vision and text we report online accuracy. For the clinical tasks we report AUROC and calibration error separately, because the two can diverge under drift. We also report the accuracy of the pseudo-labels actually used, and how far $\hat T_t$ is from the true current-model confusion matrix, which can be measured after the fact.
Decisive checks. Three outcomes would decide the proposal.
- Our method must beat both the naive delayed-label learner and the calibrated-threshold ablation at matched compute on FMoW-Time and MIMIC-IV.
- In a controlled stress test, we take late Yearbook and FMoW-Time periods and add a class-asymmetric label permutation to the drift, so that confident accuracy falls below 50%. Here we expect AnCon and PLF to degrade over the stream while our method improves. This is a falsifiable prediction that follows from the transition matrix remaining invertible.
- Plotting the gain against $d$ divided by the measured drift rate should show where the lagged estimate stops being useful. This tells practitioners when the audit is worth running.
If the scalar-calibration ablation matches the full method on all three checks, we will report that the useful contribution is recency-weighted recalibration from delayed labels, and drop the matrix.
References
- Label Delay in Online Continual Learning
- Continual Test-Time Domain Adaptation
- Less is More: Pseudo-Label Filtering for Continual Test-Time Adaptation
- Improving self-training under distribution shifts via anchored confidence with theoretical guarantees
- Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine
- Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
- Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
- Non-Exchangeable Conformal Risk Control
- Adaptive Conformal Guidance for Learning under Uncertainty
- Model Assessment and Selection under Temporal Distribution Shift
- Evaluating Model Performance in Medical Datasets Over Time