Living labs test recommender algorithms on real users: a production news portal forwards each live page view to a participant's server, and the participant must return recommendations within a hard deadline, typically 100 milliseconds. These labs exist because offline evaluation on logged data has failed to predict online results. The CLEF NewsREEL campaigns found no correlation between offline accuracy and live click-through rate. Live testing is still expensive and risky for users, so the field wants an offline replay that ranks algorithms the way a live lab would. We think today's replays miss this mainly because they strip out time. A standard chronological replay feeds events to an algorithm as fast as it can consume them and scores every answer. The live lab instead delivers events at their real arrival rate. It discards any answer that misses the deadline, and it scores each answer against a model whose state reflects only the updates it had actually finished computing.
We propose a replay protocol that puts those three timing effects back and asks whether that recovers the live ranking of algorithms. The protocol is deadline-censored, paced to the recorded arrival times, and run under a fixed memory and I/O budget. Our hypothesis is specific. Missed deadlines in a news stream are not uniform noise. They cluster in traffic bursts around breaking stories, which are exactly the moments with the highest click potential. Their cost therefore cannot be recovered by multiplying offline accuracy by an average response rate. A replay that reproduces when slow systems drop answers, and what stale state they answer from, should rank systems more like the live lab does.
Core Claim
The claim has three parts. First, a large share of the offline-to-online disconnect in stream-based news recommendation comes from resource effects that are correlated with time. These are deadline misses, update lag, and memory or I/O stalls under bursty arrival, and today's replay scoring ignores them. Second, these effects can be emulated cheaply on a single machine from logged streams. Third, the resulting deadline-censored replay metric predicts live click-through rate (CTR) rankings better than accuracy-only replay. It should also do better than the obvious correction of scaling replay accuracy by a system's average response rate.
The key quantity is the replay CTR restricted to answers that would have arrived in time, computed from the state the system would actually have had. For request $i$ arriving at logged time $t_i$, let $\ell_i$ be the measured response latency of the system under paced replay. Let $R_i(s_{t_i})$ be the recommendation list produced from the model state $s_{t_i}$ that includes only updates finished before $t_i$. Let $h(R_i)$ be 1 if a recommended item is later read by that user within the standard NewsREEL replay window. The metric is
$$ \text{dCTR} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\ell_i \le 100\,\text{ms}]\; h\big(R_i(s_{t_i})\big), $$
where $N$ is the number of requests. Accuracy-only replay drops both the indicator and the state lag. The response-rate correction replaces the indicator with its average. The two agree only if deadline misses are independent of hit probability, and our burst hypothesis says they are not.
Relation to Prior Work
The motivating evidence comes from the NewsREEL line. Benchmarking News Recommendations: The CLEF NewsREEL Use Case defines the protocol: up to six items per request, a 100ms budget, and stream replay through the Idomaar framework. CLEF NewsREEL 2016 runs 73 services from 48 teams live alongside a replay of about 100M events and reports the missing offline-online correlation. CLEF 2017 NewsREEL Overview repeats the dual live/replay design. These campaigns do record non-functional properties such as runtime and memory in replay. They report them as separate columns, though, and never fold them into an outcome metric aligned with arrival time. We make that single change, and our study is designed to test whether it closes the gap. The sharpest single example of the disconnect is CLEF NewsREEL 2017: Contextual Bandit News Recommendation. Its LinUCB meta-recommender improved offline CTR by 88.22% over random selection yet ranked 10th of 19 teams online. It also updated in batches every 300,000 messages, which is exactly the kind of update lag our metric charges for.
Streaming learning work supplies the evidence that state lag matters. Streaming CTR Prediction splits prediction and updating into interleaved phases on Avazu. It shows that choices that are good offline, such as Batch Normalization, can hurt under streaming. All of this happens in an offline script with no wall-clock deadline. Lightweight Compositional Embeddings for Incremental Streaming Recommendation shows that incremental updates taking seconds beat periodic retraining on cold items. We treat the timing of updates as part of the evaluated system rather than something to remove.
Systems work constrains the design of the emulator. Load is not what you should balance: Introducing Prequal shows on YouTube homepage serving that tail latency is driven by requests in flight rather than average CPU load. This tells us to model queueing under bursts rather than mean service time. Intermediate N-Gramming and Breaking the Storage-Compute Bottleneck in Billion-Scale ANNS show that memory-hierarchy effects, from cache and TLB misses to synchronous SSD waits, can outweigh algorithmic cost. That is why our emulator enforces a real memory cap on real hardware instead of estimating latency from operation counts.
We deliberately do not build a faster out-of-core recommender. News recommendation state in NewsREEL is small enough to fit in RAM on a modest server, so the "just add RAM" objection would defeat such a proposal. I/O enters our work as one cause of time-correlated latency that the evaluation must capture, not as the object being optimized.
Research Program
We will build a paced replay harness over the archived NewsREEL plista streams. It emits item-creation, impression, read, and request events at their logged timestamps, optionally scaled by a speedup factor. The harness wraps each participant system in a container with fixed CPU and memory quotas and optional disk-backed state. It measures response latency per request under the queueing produced by real bursts. The system's model updates run concurrently with serving, as they would live. The state used to answer a request is whatever the system has committed by that instant. The harness logs, for each request, whether the answer met the deadline and what the lag was between that request and the most recent incorporated event.
On top of this we will do three analyses. The first measures whether deadline misses concentrate in bursts and whether hit probability is higher during those same bursts. We test this directly by estimating the covariance between the timeout indicator and $h$ across request windows. If that covariance is near zero, our main hypothesis fails and we will report so. The second decomposes the gap between accuracy-only replay CTR and dCTR into three parts, attributed separately to censoring, update lag, and resource caps. The third tests prediction: does dCTR rank systems the way the live lab did?
For the systems to evaluate, we will reimplement NewsREEL baselines: recency, popularity, the BL2Beat-style baseline, and session-path heuristics. We will also reimplement the LinUCB delegation model from the 2017 bandit paper at several update batch sizes, a streaming MLP CTR model with and without Batch Normalization following the Streaming CTR setup, and an LCE-style incremental graph recommender. This spans cheap, stateless methods through to heavy, stateful ones, which is where our metric and the naive one should disagree.
Baselines
The naive reference point is accuracy-only chronological replay as used in the NewsREEL replay task. It feeds events as fast as the system accepts them and scores every answer. It is the method the field currently uses, and its failure to correlate with live CTR is the reported problem. It runs on the identical harness, the identical streams, and the identical systems, with pacing and censoring turned off. On NewsREEL we found no reported result on Lacuna for a rank correlation coefficient between replay and live CTR; CLEF NewsREEL 2016 reports only that the correlation is absent. We will therefore compute that coefficient ourselves as the reference.
The strongest competitor is the response-rate-corrected replay. It is the accuracy-only replay CTR multiplied by each system's average response rate, measured either live or in our paced harness. It is the obvious fix a reviewer would propose, and it already uses the non-functional information NewsREEL collects. It gets the same response-rate measurements our metric uses, so the only difference is whether timeouts are matched to the specific requests they affect. Beating it means a clearly higher rank correlation with live CTR, with a bootstrap confidence interval over systems and days that excludes zero improvement.
On absolute live numbers, the file gives the targets the ranking must reproduce. CLEF 2017 NewsREEL Overview reports a BL2Beat CTR of 1.17% and a top CTR of 2.09% for ORLY_KS over 42,786 impressions. CLEF NewsREEL 2016 reports typical CTRs between 0.5% and 1.0% and a best of 1.23%. For the bandit, the reported gap is an 88.22% offline gain over random alongside a 10th of 19 online rank.
Three ablations isolate the ingredients. The first keeps censoring but uses fresh state, which removes update lag. The second keeps censoring and state lag but uses uniform arrival at the mean rate, which removes burst structure. The third removes the memory and I/O cap. If the uniform-arrival ablation predicts live rankings as well as full dCTR, the burst hypothesis is wrong even if the metric works.
Why not just…?
Why not just multiply replay CTR by the live response rate? This is the most serious threat, and it is a baseline rather than something we argue around. It is correct whenever timeouts are independent of hit probability. Our burst-covariance measurement tests that assumption directly. If the covariance is negligible and the corrected replay matches dCTR in rank correlation, the honest conclusion is that the simple correction suffices. The contribution then shrinks to validating it, and we would report exactly that.
Why not just run live A/B tests, or wait for a new living lab? Live testing is the ground truth, but it costs user experience and retention, which is why Minimizing Live Experiments in Recommender Systems and the Sim4IA 2024 workshop push toward simulation. A replay that screens out systems likely to time out or lag lets a live lab spend its traffic on real contenders. Live traffic stays the judge in our validation.
Why not just build a user simulator? Simulators model how users respond to items. Our proposal models whether the answer reaches the user at all, and from what state. The two are complementary. A user simulator plugged into accuracy-only replay would still score a system that misses every burst deadline as if it had answered. The censoring ablation shows how much of the gap a better model of users could never close.
Why not just provision enough RAM and ignore resources? For a single well-engineered system this may be fine. A living lab, however, evaluates heterogeneous external participants on their own hardware. NewsREEL shows that complex models often lost to recency baselines partly through latency overheads. The evaluation must reflect the conditions participants actually run under. Our resource-cap ablation measures how much this matters. If it contributes nothing, we drop it from the protocol.
Proposed Experiments and Decisive Checks
The primary experiment is a retrospective correlation study. We reimplement the roughly ten systems above in the harness on the archived NewsREEL 2016 and 2017 streams. We compute accuracy-only replay CTR, response-rate-corrected CTR, dCTR, and the three ablations. We then compare each to live CTR and response rate. Live figures come from published values where a system corresponds to a reported team or baseline, and otherwise from a small prospective live deployment if a partner portal gives us access. Rank correlation is computed over systems, per day and pooled. Because the number of systems is small, we will report bootstrap intervals and treat the result as evidence, not proof.
A second, controlled experiment separates the causes. We hold one algorithm fixed, say LinUCB or the incremental graph recommender, and vary only resources: update batch size, memory cap forcing spill to disk, and injected burst load. Accuracy-only replay predicts no change across these variants. dCTR predicts specific drops. Each variant is scored against the censoring structure, and live traffic is used where available. This is the cleanest test, because accuracy is constant by construction.
A third check transfers the question to Avazu under the Streaming CTR protocol, with 7 days of pretraining and 3 days of streaming. Here we ask whether the reported advantage of no normalization over Batch Normalization survives once updates must finish in wall-clock time under a fixed budget. No scalar AUC for those baselines is reported on Lacuna, so we will rerun them.
The decisive result is this. dCTR must beat response-rate-corrected replay in rank correlation with live CTR, the burst-covariance must be positive, and the uniform-arrival ablation must lose most of that advantage. If all three hold, time-correlated resource effects explain a measurable part of the offline-online gap, and replay protocols should adopt this metric. If the corrected replay ties dCTR, the simpler fix wins. If neither beats accuracy-only replay, the disconnect lies elsewhere, most likely in position and interface bias, and resource faithfulness can be ruled out as its explanation.
References
- Benchmarking News Recommendations: The CLEF NewsREEL Use Case
- CLEF NewsREEL 2016: Comparing Multi-dimensional Offline and Online Evaluation of News Recommender Systems
- CLEF 2017 NewsREEL Overview: Offline and Online Evaluation of Stream-based News Recommender Systems
- CLEF NewsREEL 2017: Contextual Bandit News Recommendation
- Streaming CTR Prediction: Rethinking Recommendation Task for Real-World Streaming Data
- Lightweight Compositional Embeddings for Incremental Streaming Recommendation
- Load is not what you should balance: Introducing Prequal
- Intermediate N-Gramming: Deterministic and Fast N-Grams for Large N and Large Datasets
- Breaking the Storage-Compute Bottleneck in Billion-Scale ANNS: A GPU-Driven Asynchronous I/O Framework
- Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies
- Report on the 1st Workshop on Simulations for Information Access (Sim4IA 2024) at SIGIR 2024