Every model deployed on time-ordered data (clinical risk scores, loan defaults, flight delays) must answer a question that is usually settled informally: how much of the past should the model learn from? Keep everything and the model carries stale patterns. Keep only a recent window and it throws away statistical power. Practitioners typically compare a few retention policies on a held-out future period and report the winner. That choice is itself a leak. The policy was selected by looking at the future it is scored on, and the leak grows worse when ground-truth labels arrive weeks late, as they do in banking. We propose to make this choice online, using only labels that would actually have been available at each moment. The tool is a family of Bayesian models whose posteriors can be recomputed exactly and almost for free for every candidate window at every time step.
The enabling observation comes from discrete variational inference, meaning approximate Bayesian inference in which each parameter lives on a small grid of values (for example 4-bit integers) instead of the real line. Discretely Relaxing Continuous Variables for tractable Variational Inference (DIRECT) showed that for generalized linear models with factorized grid priors, the training objective and its gradients can be computed exactly, with no sampling noise, from a one-time precomputation over the data. After that precomputation, the cost no longer depends on the number of samples. Where that precomputation is a sum over data points, it splits into per-period pieces. A backtest over every time step and every window length then reduces to adding and subtracting stored statistics, followed by a deterministic optimizer. Our claim is that this turns retention-policy selection from a leaky after-the-fact comparison into an honest, label-delay-aware online procedure. We further claim that the exactness of the posterior matters: it makes differences between policies reflect the policies themselves rather than Monte Carlo noise.
Core Claim
Two prior findings motivate the proposal. Evaluating Model Performance in Medical Datasets Over Time (EMDOT) backtests clinical models from many historical cutoffs. It finds that the best choice between training on all history and training on a sliding window depends on the dataset and the era, and that high-capacity models degrade faster than logistic regression as data ages. On the Impact of Industrial Delays when Mitigating Distribution Drifts shows that retraining schedules tuned while ignoring labeling and deployment delays (38 days in total at BGL BNP Paribas) overstate their benefit by up to 39.86% in Matthews correlation coefficient, and that tuned schedules break when the delay changes.
Neither paper offers a mechanism that picks the policy at each moment from past evidence alone. We supply one. A retention policy $\pi = (W, \lambda)$ trains on the last $W$ periods with exponential down-weighting $\lambda \in (0,1]$ of older periods. The delay-aware prequential score of $\pi$ at time $t$ is
$$ S_t(\pi) = \sum_{s \,:\, s + d \le t} \log p\big(y_s \mid x_s,\ q_\pi^{(s - d_{\text{dep}})}\big), $$
where $s$ ranges over past periods whose labels have arrived given labeling delay $d$, $(x_s, y_s)$ are that period's inputs and outcomes, and $q_\pi^{(s - d_{\text{dep}})}$ is the discrete posterior that policy $\pi$ would have produced from data labeled before the deployment cutoff $s - d_{\text{dep}}$. At each step we deploy $\pi_t = \arg\max_\pi S_t(\pi)$. Computing this requires a posterior for every pair of policy and period. With per-period statistics $\Phi_k$, the statistic for any policy is $\Phi_\pi^{(b)} = \sum_{k=b-W+1}^{b} \lambda^{b-k}\Phi_k$. For hard windows ($\lambda = 1$) this is a prefix-sum difference, and for $W = \infty$ it is a one-line recursion. Each posterior is then a warm-started L-BFGS solve on a noise-free objective.
Research Program
The first piece of work is technical. We will verify which generalized linear model likelihoods admit DIRECT's precomputation as an additive per-period statistic. For Gaussian linear regression this holds through the usual cross-product sums. For logistic and Poisson models we will use whatever bound DIRECT applies and measure how much it loosens. The file notes that DIRECT needs Kronecker-compatible log-likelihoods and does not cover nonlinear networks, so we restrict scope to generalized linear models deliberately. EMDOT's finding that simple models are competitive under staleness makes this restriction less costly than it sounds. We will implement incremental statistics, warm starts across adjacent windows, and the selection rule above, including both labeling delay and deployment delay as in Simonetto et al.
The second piece is a diagnostic that only a discrete grid posterior gives cheaply. DIRECT reports 99.6% posterior sparsity on a 2-million-point dataset. Tracking which coefficients switch on and off as the selected window moves gives a readable account of what changed. We will check whether these switches line up with known shifts, such as COVID-era cohorts in EMDOT.
The third piece is an audit of the evaluation itself. For each dataset we compute the hindsight-optimal fixed policy, chosen by its score on the test period. This is the leaky practice, and the gap between it and our online-selected policy measures how much published comparisons of retention policies overstate. Bitstring inference, as in Approximate Bayesian Inference via Bitstring Representations (BitVI), gives an alternative discrete posterior with closed-form entropy. We include it as a second discrete family rather than the core. Its straight-through gradients are biased and do not share DIRECT's additive structure, so it serves to test whether the benefits come from discreteness in general or from exactness specifically.
Baselines
The naive reference point is a single fixed policy chosen once by convention. We run both all-history and a sliding window whose length is set from the first training block only, as in EMDOT. Our method must beat both on future predictive log-likelihood, averaged over deployment times. Only the comparison against a policy chosen from data available at deployment is fair.
The strongest existing scheduling method is the drift-triggered retraining studied by Simonetto et al. Among their 16 strategies, the error-based detector HDDM-W performed best on BGL while using 32 retrainings against 136 for periodic retraining. No per-strategy MCC values on the public Lending Club or Electricity datasets were found on Lacuna. We will rerun HDDM-W and periodic retraining ourselves under identical delays, the same model class, and the same tuning budget, with the tuning restricted to the first training block. Beating it means higher MCC and log-likelihood with no more retrainings, at every delay setting we test rather than at one tuned delay.
The strongest non-discrete competitor applies the same delay-aware prequential selection to continuous Bayesian generalized linear models. For regression this is conjugate Gaussian Bayesian linear regression, which also has additive statistics. For classification it is a Laplace-approximated logistic regression refit with warm starts per window and policy. Stochastic reparameterized Gaussian variational inference is run with matched wall-clock time. This baseline isolates the key ingredient. If it matches us, the contribution is the protocol and not the discrete inference, and we will report it that way.
A final ablation keeps DIRECT but replaces exact gradients with Monte Carlo estimates at matched compute. This tests whether noise-free scores change which policy gets selected.
For scale, the Aeolus benchmark reports TabulaRNN classification accuracy of 0.772 under a chronological 6:2:2 split, and an average AUC inflation of 0.057 from random splitting. We treat these as context, not as a target for a generalized linear model. On KOSPI 200, Pyo et al. report roll-over accuracies of 50.23% (ANN), 49.43% (polynomial SVM) and 52.90% (RBF SVM). These are near chance, so that dataset is a sanity check on our leakage audit, not a place to claim gains.
Why not just…?
Why not just retrain a continuous model on a sliding window? EMDOT shows that no single window is right across datasets or eras. Sliding windows win during abrupt shifts and lose when data is scarce. A fixed window therefore leaves the choice unmade or makes it with hindsight. Our naive baselines measure this directly. If a well-chosen fixed window ties our online selection on most datasets, the selection machinery is not worth its complexity.
Why not just use an error-based drift detector? Such detectors react only once labeled errors accumulate, and Simonetto et al. show their measured benefit collapses once realistic delays are modeled. Our selection rule also uses delayed labels. It cannot see the future any better than they can. The difference is that it compares complete candidate posteriors on the same arriving evidence instead of firing a binary retrain trigger. We test both under the same delays, so the experiment settles it either way.
Why not just use conjugate or Laplace Bayesian regression with the same selection rule? This is the real threat. For Gaussian regression, conjugate updates already give exact, additive posteriors, and discreteness adds only sparsity and non-Gaussian shape. We expect the discrete method to earn its place only on non-conjugate likelihoods and sparse or multimodal posteriors, where Laplace approximations are inexact. If the ablation shows no difference there, we will say the protocol stands on its own and the discrete inference is optional.
Why not just use reparameterized Gaussian variational inference? It works for these models, but its objective is noisy. When two policies differ by a small margin in prequential score, sampling noise can flip the selection. DIRECT's reported advantage in root-mean-squared error over reparameterization on most of its 28 datasets came from static data, so whether it carries over to the temporal setting is an open question. The matched-compute Monte Carlo ablation tests the noise argument directly.
Proposed Experiments
The datasets are the EMDOT cohorts (SEER cancer, MIMIC-IV, OPTN liver, CDC and SWPA COVID-19), the public Simonetto datasets (Lending Club, Electricity) with delays swept from zero to beyond 38 days, and Aeolus as a scale test of the claim that cost does not grow with sample count. Metrics are per-period negative log predictive density, AUROC over time, and MCC where we compare with Simonetto et al. We also report the volatility-cluster statistic from Temporal-Aware Evaluation and Learning for Temporal Graph Neural Networks, to check whether online selection shortens bursts of consecutive failures rather than only improving averages. Total compute is reported for the full backtest.
Three checks decide the outcome. First, online delay-aware selection should close most of the gap to the hindsight-optimal policy, and that gap itself quantifies selection leakage. Second, the full grid of policies across all time steps should cost within a small constant of a single training run. Third, the discrete posterior should beat the conjugate and Laplace selectors on at least the non-conjugate classification cohorts. If it does not, the paper's conclusion becomes that leakage-free retention selection is cheap with any exact Bayesian generalized linear model.
References
- Discretely Relaxing Continuous Variables for tractable Variational Inference
- Evaluating Model Performance in Medical Datasets Over Time
- On the Impact of Industrial Delays when Mitigating Distribution Drifts: An Empirical Study on Real-World Financial Systems
- Approximate Bayesian Inference via Bitstring Representations
- Aeolus: A Multi-structural Flight Delay Dataset
- Predictability of machine learning techniques to forecast the trends of market index prices: Hypothesis testing for the Korean stock markets
- Temporal-Aware Evaluation and Learning for Temporal Graph Neural Networks