Curricula That Can Tell Hard Data From Stale Data

When a model is trained on data collected over years and deployed in the future, the old records pose a problem. Some old examples are simply hard and still worth learning. Others are hard because the world changed: a clinical definition was revised, or a news category drifted, so the old label now contradicts what newer data says. Adaptive curricula, meaning training schedules that change which examples the model sees based on how it is currently doing, almost always treat "high loss" or "low confidence" as "not yet learned, show it more." Under temporal drift that rule is backwards for stale examples: it steers training toward the very data that pulls the model away from the future.

We propose a closed-loop curriculum that judges historical data by a different criterion. It asks whether training on this group of past examples currently lowers loss on the most recent labeled period available before deployment, not whether the group is hard. The scheduler reweights groups defined jointly by time slice and content cluster. It updates those weights online from cheap gradient-agreement measurements, and it respects strict chronological splits and labeling delays. The claim to test is that this signal keeps old data where the underlying concept is stable and drops it where the concept has changed. Neither pooled training, a fixed recent window, nor a single age-decay weight can do both at once.

Core Claim

The file shows a tension that no existing method resolves. On MIMIC-IV, Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine evaluates invariance-seeking and adaptation methods across four 3-year blocks and finds none beats plain training on the largest pooled history. On financial streams, On the Impact of Industrial Delays when Mitigating Distribution Drifts finds pooling all history counterproductive compared with a recent sliding window. Class-Incremental Experience Replay for Continual Learning Under Concept Drift shows that replaying uncurated old data after concepts flip collapses accuracy to 50–60%. The same memory kept above 91% once it prunes and relabels impure clusters.

Our reading is that "how much history to use" is the wrong unit of decision. Staleness is local: it affects some concepts and time ranges but not others. The right unit is a (time slice, content cluster) group, and the right signal is whether that group's gradient currently agrees with the gradient of the most recent labeled window.

This is a claim about the curriculum signal, and it makes a sharp, falsifiable prediction. Standard difficulty-driven schedulers will systematically upweight pre-shift examples on drifting concepts and hurt future performance. Such schedulers include self-paced weighting as in Self-Paced Learning for Neural Machine Translation, high-loss prioritization as in Loss Max-Pooling for Semantic Image Segmentation, and the "keep examples near 50% accuracy" rule of CurES. The agreement signal should do the opposite. If difficulty-based curricula do not show this failure on real chronological benchmarks, the motivation for the proposal weakens considerably, and we will report that.

Research Program

The method borrows its update rule from Skill-it!. That work reweights groups of training data by online mirror descent, a multiplicative update that raises the weight of groups that help a target. It targets a fixed set of skills whose dependency graph is estimated in expensive upfront runs. We change the target and the groups.

The target is a probe set: the most recent labeled window before the training cutoff. It is separated from the training slices by a buffer gap, as in the ENSO setup of Temporal Convolutional Networks for the Advance Prediction of ENSO. When labels arrive late, the probe is shifted back by the labeling delay, following On the Impact of Industrial Delays when Mitigating Distribution Drifts.

The groups are time slices crossed with k-means clusters of penultimate-layer embeddings from a short ERM warm-start. As an ablation, we will also cluster by loss trajectories, which Skill-it found better reflects what a model is actually learning.

Let $w_g$ be the sampling weight of group $g$. Let $\nabla L_{\text{probe}}$ be the loss gradient on a probe minibatch, and $\nabla L_g$ the gradient on a minibatch from group $g$. Every $m$ steps we update

$$w_g \leftarrow \frac{w_g \exp\left(\eta \, \langle \nabla L_{\text{probe}}, \nabla L_g \rangle / \lVert \nabla L_{\text{probe}} \rVert\right)}{\sum_{g'} w_{g'} \exp\left(\eta \, \langle \nabla L_{\text{probe}}, \nabla L_{g'} \rangle / \lVert \nabla L_{\text{probe}} \rVert\right)},$$

where $\eta$ is a step size. The inner product is the first-order estimate of how much one step on group $g$ would reduce probe loss. The update is computed on the last layers only, so it costs a few extra backward passes per update, not extra training runs. Weights are floored at a small value so that no group is permanently discarded on noisy evidence.

The probe is small, so it is noisy and can be overfit. We therefore choose how far back the probe window reaches using the Goldenshluger–Lepski rule from Model Assessment and Selection under Temporal Distribution Shift. That rule grows an evaluation window while the recent past looks stationary and shrinks it when it does not. In that paper the rule served only post-hoc model selection. Here it sets the target that a training-time scheduler aims at.

The analysis half of the program has two parts. First, we log group weights against time slice to test whether the learned weighting is non-monotone in age: old stable clusters kept while recent-but-shifted clusters are dropped. Second, we locate where difficulty-based curricula and the agreement signal disagree, and check those disagreements against known change points. The clearest known change point is the Sepsis-3 definition revision in MIMIC-IV.

Baselines

The naive reference is pooled ERM with uniform minibatches over all pre-cutoff data. Both Wild-Time and Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine find it matches or beats every invariance and continual-learning method they test, which also makes it the strongest existing method on these benchmarks. Wild-Time reports a drop from about 98% in-distribution to about 79% out-of-distribution accuracy on Yearbook, a drop in mortality AUC from 0.90 to 0.72 on MIMIC-IV, and a 20% average gap across its datasets. The file does not attribute these figures to a specific method, so we will rerun ERM ourselves with Wild-Time's protocol. Guo et al. report sepsis AUROC decaying by up to 0.090 across blocks.

The second baseline is a fixed recent sliding window, tuned over window length. Simonetto et al. found this beats pooling on financial data.

The third is exponential age-decay weighting, with the decay rate tuned on the same probe. It is the most dangerous baseline because it uses the same recency information as our method with one scalar. We found no reported result on Lacuna for exponential age-decay weighting on Wild-Time.

The fourth is two-stage training: pooled ERM followed by fine-tuning on the probe window.

The fifth is a family of difficulty-based closed-loop curricula applied over the pooled history. It comprises self-paced confidence weighting; an item-response-theory schedule in the style of Dynamic Data Selection for Curriculum Learning via Ability Estimation, which scores each example's difficulty and admits only examples the model's estimated ability can handle; and the 50%-accuracy targeting of CurES, adapted to classification by predicted-probability targeting. These are the closest methodological competitors. None of them has been evaluated on chronological splits, so no reported result exists on Lacuna for them there.

All methods get the same backbone, the same number of gradient steps, the same pre-cutoff data, and the same hyperparameter search budget of 20 configurations. Selection uses only the probe window; no post-cutoff data is used for tuning.

Two ablations isolate the key ingredients. The first restricts groups to time slices alone, with no clusters. If this matches the full method, content-local staleness is not doing the work. The second replaces the agreement signal with per-group loss inside the identical update rule. This isolates the signal from the machinery.

Beating the field means a statistically reliable improvement over the best of tuned age-decay, sliding window, and two-stage fine-tuning on held-out future years. Reliability is measured across at least five seeds, on at least three of the five Wild-Time datasets. On MIMIC-IV sepsis, the improvement must appear in AUROC rather than only in calibration.

Why not just…?

Why not just pool all history and train ERM? On most Wild-Time datasets this may be very hard to beat, and that is exactly what we test. The file's evidence of harm from pooling comes from financial streams and synthetic label flips, not from Wild-Time. Our prediction is concrete. Gains should concentrate where there is a documented concept change, such as MIMIC-IV sepsis across the Sepsis-3 revision, and should be near zero on slowly drifting data such as Yearbook. If gains are near zero everywhere, pooled ERM wins and the paper reports that.

Why not just weight examples by age with a tuned exponential decay? This is the real threat. A single decay rate captures "recent matters more," and if drift is roughly uniform across concepts it could match our method. It cannot keep old data on a stable concept while dropping newer data on a changed one. The time-slices-only ablation and the tuned-decay baseline together settle whether that capacity matters. If tuned decay ties the full method, we will conclude that staleness on these benchmarks is not content-local.

Why not just fine-tune the pooled model on the most recent window? Fine-tuning adapts to the latest period but tends to forget, and it cannot correct representations already shaped by contradicting old data. The systematic review Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine found that output-level fixes such as recalibration restore calibration but not discrimination. Only expert code remapping preserved discrimination in that review. Recalibration is not fine-tuning, though, so the review does not settle this alternative. We include two-stage fine-tuning with a tuned step budget and compare AUROC directly.

Why not use an existing adaptive curriculum off the shelf? Our central hypothesis is that difficulty signals mistake stale examples for unlearned ones. The loss-signal ablation and the difficulty-curriculum baselines test this directly. If they perform as well as the agreement signal, the hypothesis is wrong.

Proposed Experiments

The main evaluation uses Wild-Time's Eval-Fix protocol on Yearbook, FMoW-Time, HuffPost, arXiv, and MIMIC-IV: train before a cutoff year and test on each subsequent year. Accuracy is plotted against time distance so we can see whether our method flattens the decay curve rather than just shifting it. The Guo et al. setup adds MIMIC-IV at 3-year-block granularity for four tasks, reporting both AUROC and calibration error. Its sepsis task is the pre-registered decisive test: the method should down-weight pre-2016 sepsis-positive clusters and recover AUROC that recalibration cannot.

The second setting uses Lending Club and Electricity, evaluated with Simonetto et al.'s labeling and deployment delays. The probe is lagged accordingly, testing whether the closed loop survives feedback that is weeks old; every curriculum method in the file assumes immediate feedback.

The diagnostic experiment runs first and is cheap. On MIMIC-IV and HuffPost, we train self-paced and CurES-style schedulers and measure how much of their weight lands on pre-shift slices. We then correlate each group's loss-based weight with its agreement weight. A strong negative correlation on drifting clusters confirms the premise. A positive correlation means difficulty and staleness are not confounded in practice, and the project would pivot to the simpler question of whether any closed-loop curriculum helps under chronological splits.

The final check is to run the full method and all baselines under Wild-Time's Eval-Stream protocol, where the model is updated incrementally. We track group weights over successive retrainings to test whether the learned weighting follows known shifts without being told when they happen.