INTENT: Out-of-Sample, Out-of-Time, and Cold-Start Evaluation on User Event Sequences

Mattias Kvist, Erik Lidman Hillbom, Edoardo de Cal, Davis Siemens, Adam Fredriksson, Rickard Brüel-GabrielssonarXiv 2026 · arXiv

TL;DR

The INTENT benchmark introduces a rigorous evaluation framework for sequential recommendation models, moving beyond flawed "leave-last-out" protocols to a strict global time split. By testing models across a 7-cell grid of seen/new users, items, and time periods, the authors reveal that current state-of-the-art transformers fail completely on new items and that predicting the future is significantly harder than generalizing to new users.

What Problem Does This Paper Address?

In fields like banking and e-commerce, "foundation models" are trained on chronological sequences of user events (transactions, clicks, etc.). When deployed, these models must predict the future: what will an existing customer do next month? What will a brand-new customer do at a merchant that didn't even exist during training?

The authors identify a major "evaluation crisis" in sequential recommendation research:

  1. Data Leakage: 77% of recent papers use a "leave-last-out" protocol, which holds out the final interaction for each user. Because users have different timelines, training data often contains events that occur after test events, allowing models to effectively "learn from the future."
  2. Lack of Realism: Most benchmarks don't separately measure performance on new users (cold-start) or new items (novelty) in a future time period (out-of-time).
  3. Data Quality: Many public datasets lack the precise calendar timestamps required to enforce a true chronological split.

Methods

The core of the paper is the INTENT evaluation grid. It categorizes every test event into one of seven cells based on three binary axes:

  • User: Is the user known from training, or are they a 10% held-out "new" user?
  • Time: Did the event happen before or after a fixed per-dataset cutoff date?
  • Item: Had the target item appeared in the training set?
Figure 1: The seven test cells, laid out as in Figure 4. Rows sep- arate training users from a 10% held-out sample, and columns separate events before and after a per-dataset cutoff. Each row and period is divided into seen and new target items, except the grey cell, which repeats training events and so holds only seen items. Shading counts how many of the user, time, and item axes are new to the model.

To ensure the benchmark is robust, the authors conducted a massive audit of 59 public dataset families. They enforced a strict "global time split" rule: roughly 80% of all events must precede the cutoff date. This ensures that no training event follows any test event in the "future" cells.

Datasets

After auditing 101 public releases, only five datasets met all criteria (stable entities, calendar timestamps, >25k users, clear action mapping, and accessibility).

Figure 2: Of 59 dataset families, 36 pass automated checks of ordering, population, and actions, and 5 are selected. The largest group of exclusions meets every criterion but duplicates a condition already covered. Ordering is the most frequent defect: 19 families have no calendar time that orders events.

The selected "Big Five" are:

  • Amazon H&K: Sparse data spanning 25 years.
  • KuaiRand: Includes random exposure and explicit negative feedback.
  • MicroLens: Rich item content (video, text, audio).
  • H&M: Includes static customer profiles and prices.
  • REES46: A multi-category "purchase funnel" (view → cart → buy).

Key Insights and Ideas

  • "Time costs more than a new user": Models generally perform similarly on known and new users within the same time period. However, crossing the "time cutoff" into the future causes a massive performance drop (25% to 68% for the SASRec model). This suggests that shifts in global behavior over time are a much bigger challenge than individual user cold-starts.
  • The Memorization Gap: Standard identifier-based models like SASRec score zero on new items. They have effectively "memorized" the catalog.
  • Metadata is not a silver bullet: Adding item metadata (SASRec-F) only significantly helped on the content-rich MicroLens dataset. On others, it barely moved the needle for new items.
  • Repeat-Last is a powerful baseline: Simply predicting that a user will repeat their last action is often the best strategy for "new items" (items the model hasn't seen, but the user has interacted with earlier in the test period).

Key Results

The authors evaluated four models: a Popularity baseline, a "Repeat-last" control, the SASRec transformer, and a metadata-aware variant (SASRec-F).

Figure 4: NDCG@10 (×100, mean over users) of popularity (Pop), repeat-last (Rep), SASRec (SAS), and SASRec-F (SAS-F), laid out as in Figure 1. Bold marks the best model in a cell when its 95% confidence interval overlaps no other model’s. Red outlines mark datasets below the cell’s user floor; their scores do not rank models. Shading is log-scaled to each dataset’s best score over three decades, and white is zero. Catalogue sizes differ by a factor of 350, so scores are not comparable across datasets.
  • SASRec dominates on "seen items" in the future, often doubling the performance of popularity-based baselines.
  • Repeat-last wins on "new items" in four out of five datasets. This highlights that a user's recent history is often more predictive of a "new-to-model" item than any learned representation.
  • On MicroLens, SASRec-F was the only model to achieve a non-zero score on new items for new users, thanks to the rich video metadata.

Limitations and Future Work

  • Platform Confounding: Because each unique condition (like "purchase funnel") is tied to a specific dataset, it's hard to tell if a result is due to the condition or the platform itself.
  • Untuned Models: The reference models were used with standard hyperparameters; tuning might shift the rankings.
  • Public vs. Private Data: No public dataset perfectly mirrors a bank's ledger. The authors found that existing "public banking datasets" often fail the ordering criterion due to anonymization noise.
  • Future Directions: The authors suggest that future work should focus on models that can ingest knowledge from outside the training split, such as pretrained content encoders or cross-platform transfer learning.

Summary updated

Abstract

Banks now train foundation models on ordered sequences of customer events such as transactions, app sessions, and authorizations. Deployed, such a model predicts forward in time, often for customers and merchants absent from training, yet each is evaluated on private data, so no two can be compared. Public benchmarks for such sequences rarely test a period the model has not seen; recommendation studies usually hold out each user's last events instead. We present INTENT (INTEraction, Novelty, and Transfer evaluation for next-action prediction), a public benchmark that scores a model in seven cells defined by whether the user, the period, and the target item appeared in training. An audit of 59 public dataset families retains five datasets; the most common defect is the lack of a calendar timestamp that orders events. No date lies within the time span of all five datasets, so each is split at its own cutoff, chosen by one rule fixed before any model is evaluated. Summed over the datasets, every cell holds at least 111,769 users. In reference results the cell determines the best model: a transformer leads on seen items after the cutoff but scores zero on new items, a variant that embeds item metadata helps mainly on MicroLens, and repeating the user's last item leads on new items in four of five datasets. This parameter-free control also shows that on three datasets most of the score lost after the cutoff reflects changed behavior rather than the model. The pipeline, the cutoff rule, and the per-cell user counts are publicly available.
BibTeX
@misc{kvist2026intent,
  author = {Mattias Kvist and Erik Lidman Hillbom and Edoardo de Cal and Davis Siemens and Adam Fredriksson and Rickard Brüel-Gabrielsson},
  title = {{INTENT: Out-of-Sample, Out-of-Time, and Cold-Start Evaluation on User Event Sequences}},
  year = {2026},
  eprint = {2611.13069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13069},
}
Figure