When a model is deployed on data that changes over time, the labels usually arrive late. A bank learns whether a transaction was fraud weeks after it happened. As a result, at any moment the newest labeled data is stale while the newest unlabeled inputs are already on hand. This note proposes to use those recent unlabeled inputs as a target. We reweight the old labeled examples so that, as a group, they look like the inputs seen during the labeling delay. The reweighting uses kernel covariate balancing, which chooses example weights so that the weighted training set matches a target set on a very rich family of nonlinear summaries rather than only on means and variances. The payoff comes from a property that follows from how the problem is set up. The delay window is the one piece of "future-looking" data that is legitimately available without peeking ahead.
The question matters because chronological evaluation keeps showing that models look better than they are. Evaluations that pretend labels arrive instantly overstate how well drift mitigation works by up to 39.86% on real banking data, where the combined labeling and deployment delay is 38 days (On the Impact of Industrial Delays…). Kernel reweighting methods that could correct the resulting covariate shift assume batch access to the test inputs. That assumption becomes look-ahead bias under a strict time split. This proposal resolves that tension through the choice of target rather than through a new estimator.
Core Claim
Kernel balancing toward the covariates that arrive during the labeling delay should improve post-deployment accuracy over unweighted retraining and over the standard recency heuristic of exponential time-decay weights. The gains should come mainly from streams with recurring drift. In those streams, balancing by input content can raise the weight of an old regime that has come back, which no weighting based on age can do. On streams with monotone drift, we expect balancing to roughly tie with time-decay, and we will report that outcome if it occurs.
Concretely, at deployment time $t$ with delay $\delta$:
- The labeled set is $L=\{(x_i,y_i): t_i \le t-\delta\}$, with $n$ points.
- The unlabeled target is $U=\{x_j: t-\delta < t_j \le t\}$, with $m$ points.
We solve
$$ \min_{w \ge 0,\ \frac{1}{n}\sum_i w_i = 1}\ \Big\| \tfrac{1}{n}\textstyle\sum_{i \in L} w_i \Phi(x_i) - \tfrac{1}{m}\sum_{j \in U}\Phi(x_j) \Big\|_{\mathcal{H}}^2 + \lambda \|w\|_2^2 , $$
where:
- $\Phi$ is the feature map of a Gaussian kernel with bandwidth $\sigma$;
- $\mathcal{H}$ is the associated function space, a reproducing kernel Hilbert space;
- $\lambda$ penalizes weight variance, as in Kernel Optimal Weighting.
The first term is the maximum mean discrepancy (MMD), a distance between the two distributions measured in that space. Kallus showed that this term equals the worst-case bias when the outcome function lies in a bounded ball of $\mathcal{H}$ (A Framework for Optimal Matching for Causal Inference). A double-weighted variant also puts weights on $U$, so the method can discount target points that fall outside historical support.
Research Program
Method components. The method is built from existing parts, and the new element is where the target comes from.
- The balancing objective is the kernel mean matching and dual-norm formulation of Kallus, used here for covariate shift across time rather than for treated-versus-control balance.
- The double-weighted variant follows DW-KMM from Double-Weighting for Covariate Shift Adaptation. That work weights training and test points jointly to tame exploding ratios, but it treats the test inputs as a batch handed over transductively. We restrict the target to inputs whose timestamps precede the decision, so the transductive step becomes honest under a strict chronological split.
- Scale is handled with random Fourier features, a finite random approximation to the kernel. The objective then becomes a $D$-dimensional matching problem solvable by projected gradient. This is the same approximation FCore uses (Streaming Covariate Balancing via Discrepancy-Based Feature Coresets), and it follows the large-data kernel balancing line (A kernel balancing approach that scales to big data).
How this differs from FCore. FCore balances treated against control units inside a stream and assumes that outcomes arrive instantly. We balance past against present and take the delay as the defining constraint.
Tuning without look-ahead. Bandwidth sensitivity is a known open problem for these estimators (Kallus & Santacatterina). We choose $\sigma$ and $\lambda$ by a nested backtest that stays inside the labeled history. We pretend the cutoff was $t-\delta-\Delta$, treat the following $\delta$ as the delay window, and score on the next $\Delta$ period. Every tuning decision uses only data the deployed system would actually have had.
Diagnostics. Covariate balancing cannot repair a change in the label rule $p(y\mid x)$. For each deployment step we therefore measure two things:
- MMD between delay window and deployment window. This shows how good a stand-in the delay window is for the period the model will actually face.
- Concept-shift gap. This is the accuracy difference between our reweighted model and a model trained on true deployment-period labels, a look-ahead oracle. It estimates how much of the remaining error comes from concept drift, which reweighting cannot fix.
These diagnostics tell us when the method should and should not help. They also serve as the direct empirical test of the static-RKHS-ball assumption criticized in the research file.
Baselines
Unweighted retraining. The naive reference retrains the same classifier on all labeled history up to $t-\delta$ at each retraining point, following the delayed-evaluation protocol of Simonetto et al.. Everything else is judged relative to this. Its value on these benchmarks is as a number we measure ourselves: no reported result found on Lacuna for per-dataset MCC (Matthews correlation coefficient) on LCLD or Electricity under delay.
Exponential time-decay weighting. This is the strongest simple competitor. It already puts more weight on recent data and has one parameter, the half-life. We tune the half-life with the same nested backtest and the same number of configurations as our $(\sigma,\lambda)$ grid.
Drift-detector-triggered retraining. The strongest existing method on this question is HDDM-W retraining, as evaluated by Simonetto et al. They report that it needs 32 retrainings against 136 for aggressive periodic retraining and that it sits on their effectiveness–cost Pareto front. The file reports no MCC value for it, so we rerun it under the identical delay. We also test whether our weights compose with it by applying our reweighting at each HDDM-W-triggered retrain.
Domain-classifier density ratio. This parametric route fits a logistic regression separating $L$ from $U$ and uses its odds as weights. It is the parametric IPTW analogue that Kong et al. found to have roughly twice the error of MMD balancing under nonlinear confounding on Kang-Schafer.
Single-sided versus double-weighted KMM. This ablation isolates the double-weighting ingredient. As an implementation check, we first reproduce DW-GCS on 20 Newsgroups "comp vs talk." There Segovia et al. report a test error of 0.11 against 0.34 for KMM.
Look-ahead oracle. The key ablation replaces the delay-window target $U$ with the true deployment-window inputs, which is forbidden in practice. The gap between the oracle and our method measures what is lost by using the delay window as a stand-in.
What counts as winning. All methods share the same base learner (gradient-boosted trees and logistic regression), the same retraining points, and the same tuning budget. Beating the baselines means a higher mean MCC than both time-decay and HDDM-W. The comparison is paired across retraining points with a sign test at $p<0.05$, pooled across all four streams. On the recurring-drift streams specifically, we expect a margin that survives this test.
Why not just…?
Why not just use exponential time-decay weights? This is the real threat. On streams where drift only moves forward, recency and covariate similarity largely coincide, and we expect a tie. The case for balancing rests on recurring regimes such as electricity-demand seasons and the recurring-drift segments of INSECTS. The experiment settles it by reporting results separately for recurring and monotone segments. If balancing does not beat time-decay on recurring segments, the claim fails.
Why not just retrain on the most recent window, or retrain when a detector fires? Retraining changes when to update, while balancing changes how the available data is weighted, so the two are complementary. Simonetto et al. show that under delay, periodic retraining is costly and still degrades. Detector-triggered retraining is cheaper but still trains only on stale labels. We test the combination directly. If HDDM-W alone matches HDDM-W plus balancing, the weights add nothing.
Why not just a logistic-regression density ratio? It is cheap and assumption-light on its face. However, it is misspecified whenever the shift is nonlinear, and its odds explode under support mismatch. Both failure modes are documented by Kong et al. and Segovia et al.. It runs as a baseline with the same tuning. If it matches the kernel method, we conclude that the target choice, not the kernel, is doing the work, and we will report that plainly.
Why not just apply DW-KMM as published? That is largely what we do. The contribution lies in the protocol and target, not in a new solver. Published DW-KMM uses the test inputs themselves, which under a chronological split is look-ahead. The look-ahead oracle arm is exactly that setup, and our method is its honest counterpart.
Proposed Experiments
Datasets and protocol. We use the public streams from the delay study: Lending Club Loan Data (1.1M loans over 5 years) and Electricity (45k samples over 943 days). We add the INSECTS streams from MCD-DD, chosen because they contain labeled recurring drift. We will request BGL BNP Paribas access but do not depend on it. For each stream we impose delays $\delta$ spanning none, a moderate lag, and a lag proportional to the 38-day banking figure relative to stream length. Metrics are MCC, following Simonetto et al., and accuracy. For each retrained model we also record the effective sample size of the weights.
Decisive checks.
- Does balancing work at all? It must beat unweighted retraining and time-decay under nonzero delay. At zero delay the target window is empty, so the method must reduce to unweighted retraining. This provides a built-in sanity check.
- Does the delay window work as a stand-in? It should recover a substantial fraction of the oracle's gain over unweighted retraining. That fraction should shrink as the MMD between the delay window and the deployment window grows. If this dose-response relationship does not appear, the mechanism is not what we claim.
- Where do the gains come from? Gains should concentrate on recurring-drift segments.
- When should the method fail? On segments where the concept-shift gap dominates, no weighting method should help, and ours should degrade no worse than unweighted retraining. Double-weighting is designed to guarantee this by damping extreme weights. Failing this check would argue against deploying the method without the diagnostic.
References
- On the Impact of Industrial Delays when Mitigating Distribution Drifts: An Empirical Study on Real-World Financial Systems
- A Framework for Optimal Matching for Causal Inference
- Causal Inference by Minimizing the Dual Norm of Bias: Kernel Matching & Weighting Estimators for Causal Effects
- Double-Weighting for Covariate Shift Adaptation
- Streaming Covariate Balancing via Discrepancy-Based Feature Coresets
- A kernel balancing approach that scales to big data
- Optimal balancing of time-dependent confounders for marginal structural models
- Covariate balancing using the integral probability metric for causal inference
- Online Drift Detection with Maximum Concept Discrepancy