Calibrated Synthetic Twins for Diagnosing Preference Drift in Recommender Graphs

Recommender systems learn from logs of who interacted with what and when. People's tastes change over time, and a model that cannot follow those changes keeps recommending what a user liked months ago. The trouble is that no one can tell, from real logs alone, why a model fails on them. A user who suddenly clicks unfamiliar items may have changed taste, may be reacting to new items in the catalog, or may simply have misclicked. We propose to build synthetic twins of real interaction datasets. These are generated user–item graphs, fitted to match a specific real dataset, in which the drift in each user's taste, the turnover of the catalog, the habit of re-consuming the same items, and click noise are separate knobs with known ground truth. A twin only counts as valid if it reproduces how competing methods rank on the real data. Once validated, it lets us take apart the failures that real benchmarks can only expose.

The opportunity is concrete. TGB-Seq showed that temporal graph neural networks (models that update node representations as timestamped edges arrive) do well on standard benchmarks mostly by memorizing repeated edges. Their accuracy drops by a factor of 4x to 8x on interactions never seen before, and plain sequential recommenders beat them on near-zero-repeat datasets such as ML-20M and GoogleLocal. Because TGB-Seq is built entirely from observed data, it cannot say whether that collapse comes from drifting taste, catalog churn, or something else. A second unresolved tension concerns noise removal. DeBaTeR prunes interactions that do not fit a user's history, while the drift literature suggests that genuine taste changes begin with exactly such interactions. Both questions need ground truth that only a generator can provide, and the generator has to be close enough to real data for its answers to carry over.

Core Claim

The core claim is that a time-varying, degree-corrected mixed-membership bipartite generator, fitted per dataset, can serve as a trustworthy stand-in for real recommendation logs. The test is that method rankings on the twin agree with rankings on the real dataset. On validated twins, two things become measurable. First, the unseen-edge collapse documented by TGB-Seq can be split into separate contributions from drift rate, catalog turnover, and repeat propensity. Second, the rate at which noise-pruning methods discard edges at the onset of real drift can be measured directly.

What is new here is not synthetic graphs for graph learning. GraphWorld already sweeps static graph properties and finds ranking phase transitions near 0.5 edge homogeneity, a region real benchmarks barely cover. Synthetic temporal diagnostics are not new either: T-GRAB tests periodicity, delayed memory, and multi-hop propagation on abstract homogeneous graphs. The new elements are three. The generator is bipartite and built around user preferences. It has independently controllable drift and turnover. And its realism is validated by whether method rankings agree, so the twin is not merely a plausible-looking graph.

The Generator

Direction 1 supplies the backbone. Network Goodness-of-Fit for the Block-Model Family found that the degree-corrected mixed-membership model (DCMM) is the simplest block model that fits real networks well. In a DCMM, each node has an activity level and a soft membership over several latent communities. On Citee2016 its goodness-of-fit statistic is 3.687, against 759.2 for a plain stochastic block model. That result constrains our design: a generator lacking degree correction or mixed membership would already be wrong on static structure. We therefore extend DCMM to time. At time $t$, user $u$ with activity $\theta_u$ and taste vector $\pi_u(t)$ interacts with item $i$ (popularity $\theta_i$, profile $\pi_i$, alive during window $[a_i, d_i]$) at rate

$$ \lambda_{ui}(t) = \theta_u\,\theta_i\,\pi_u(t)^\top B\,\pi_i \cdot \mathbf{1}[a_i \le t \le d_i] \cdot \big(1 + \rho\, r_{ui}(t)\big), $$

where $B$ is a community affinity matrix, $r_{ui}(t)$ counts the user's past interactions with $i$, and $\rho$ is a repeat-consumption knob. Uniform random edges are added at rate $\epsilon$ to model click noise.

Drift lives entirely in the trajectory $\pi_u(t)$. We implement four schedules on the simplex. Gradual drift is a slow random walk. Sudden drift is a jump at a per-user change point. Periodic drift is seasonal oscillation. Triggered drift is a jump that fires only after the user is exposed to an item from a dormant community, imitating the triggered preferences that AcCoRD found frontier agents nearly unable to elicit (scores as low as 0.09). Drift speed varies across users, following the finding in DPS that per-node decay rates beat a single global rate. Catalog turnover is set by the item lifetimes $[a_i, d_i]$, independently of $\pi_u(t)$. This separation is the property that real logs lack.

Research Program

The work proceeds in three stages. First, we fit the generator to each bipartite TGB-Seq dataset (ML-20M, Taobao, GoogleLocal) and to the DeBaTeR datasets (ML-1M, Yelp, Amazon Movies & TV). Item lifetimes and activity levels are estimated directly from the data. Memberships and $B$ come from mixed-membership spectral estimation on successive time windows, and drift schedule parameters are chosen by matching the moments of membership change between windows. Second, we validate each twin at two levels. At the structural level, the twin should reproduce repeat ratio, degree distributions, inter-event-time distributions, and windowed goodness-of-fit. The windowed goodness-of-fit check applies GoF-MSCORE to each time slice of the user–user co-consumption projection, and we treat it only as a heuristic, because projection breaks the edge-independence assumption the statistic's theory rests on. At the functional level, the twin should reproduce how a fixed set of methods ranks against each other on the real data. Third, using only validated twins, we run controlled sweeps over drift rate, schedule type, turnover, repeat propensity $\rho$, and noise $\epsilon$. Each generated edge is labeled as noise, repeat, within-taste, or drift-onset, which makes direct diagnosis possible.

Baselines

The methods under study run under the TGB-Seq protocol: mean reciprocal rank against 100 sampled negatives, reported separately for seen and unseen edges. Every method gets the same chronological splits, the same hyperparameter search budget (an equal number of trials per method, tuned on the twin's validation split and separately on the real validation split), and the same hardware cap. Methods that exceed the cap are reported as failures, as TGB-Seq reports for DyGFormer.

The naive reference point is a persistence and popularity predictor in the spirit of EdgeBank, which T-GRAB evaluates. It recommends what the user has done before, padded with globally popular items. Any method that cannot beat it on unseen drift-onset edges is not tracking drift at all. The static reference is LightGCN, which aggregates all history with no notion of time and is DeBaTeR's own baseline. The strongest existing competitor on near-zero-repeat bipartite data is SGNN-HN, which TGB-Seq reports as significantly outperforming temporal graph networks on GoogleLocal and ML-20M. No reported result found on Lacuna for SGNN-HN's exact MRR on these datasets, so we rerun it ourselves under the shared protocol. We also include SASRec as the standard self-attention sequence model, TGN and DyGFormer as temporal graph networks, DPS for personalized decay, and DeBaTeR-A and DeBaTeR-L for denoising.

At the benchmark level, the competitors for the twins are a static GraphWorld-style generator with the same fitted DCMM but frozen $\pi_u$, and semi-synthetic injection into real logs. The ablation that isolates our key ingredient fits two twins that agree on aggregate statistics, one without calibrated per-user drift and one without the separate turnover process. If either still reproduces real rankings, drift modeling is unnecessary, and we will say so. The twin beats its competitors if the Kendall rank correlation between method rankings on twin and on real data is clearly higher than for the static and semi-synthetic alternatives, measured across held-out datasets that were not used during generator design.

Why not just…?

Why not just inject drift into real logs, as NS-DPO does with preference flips? NS-DPO builds non-stationary benchmarks by flipping labels at a change point. That was enough to show standard DPO's reward accuracy falling to roughly 50–55% while time-discounted training held above 60–70%. The equivalent for graphs, swapping parts of user histories at change points, keeps real structure and costs little, so it is a real threat. Its weakness is that real logs already contain unknown drift and turnover, so the injected labels sit on contaminated ground truth, and turnover cannot be varied separately. We keep injection as a competing benchmark. If its method rankings and its decomposition results agree with the twins', the cheaper tool wins for those questions, and we will report that.

Why not just stratify TGB-Seq's unseen edges by observable proxies, such as a user moving to a new item category? Proxies of this kind mix drift with turnover: a category shift may happen only because the old items left the catalog. On twins we can compute such a proxy and measure its precision and recall against the true drift-onset labels. If the proxy recovers drift well, practitioners can apply it directly to real data, which would itself be a useful finding.

Why not just extend T-GRAB with bipartite tasks? T-GRAB's tasks are hand-designed reasoning probes on abstract graphs, and nothing in them is fitted to any real dataset. It already showed that models crash at change points under a persistence heuristic, but it cannot say whether that matters on ML-20M. Calibration together with ranking validation is the missing piece, and the static-twin ablation tests directly whether that piece is needed.

Decisive Experiments

The first experiment checks fidelity. On held-out datasets, does the calibrated drifting twin reproduce real method rankings better than the frozen-membership and injection alternatives? If not, the program stops at a negative result about how far synthetic recommender benchmarks can be trusted.

The second experiment decomposes the unseen-edge collapse. Sweeping drift rate, turnover, and $\rho$ independently, we fit how the unseen-to-seen MRR ratio of TGN and DyGFormer changes along each axis. We then ask which knob, set to its calibrated value, reproduces the 4x–8x gap that TGB-Seq reports. We expect repeat propensity and drift to account for most of it, but the experiment is designed so that turnover could turn out to dominate.

The third experiment tests the pruning paradox. For DeBaTeR, we measure the fraction of its pruned or down-weighted edges that are true drift-onset edges rather than injected noise, as $\epsilon$ and drift speed vary. We also measure how MRR in the window after each user's change point compares with a version that prunes only true noise. A high drift-onset share in the pruned set, combined with an MRR deficit after change points, would confirm that current denoising removes the very signal of changing taste. That result would motivate denoisers that treat an out-of-pattern edge as noise only after it fails to recur.