TL;DR
The INTENT benchmark introduces a rigorous evaluation framework for sequential recommendation models, moving beyond flawed "leave-last-out" protocols to a strict global time split. By testing models across a 7-cell grid of seen/new users, items, and time periods, the authors reveal that current state-of-the-art transformers fail completely on new items and that predicting the future is significantly harder than generalizing to new users.
What Problem Does This Paper Address?
In fields like banking and e-commerce, "foundation models" are trained on chronological sequences of user events (transactions, clicks, etc.). When deployed, these models must predict the future: what will an existing customer do next month? What will a brand-new customer do at a merchant that didn't even exist during training?
The authors identify a major "evaluation crisis" in sequential recommendation research:
- Data Leakage: 77% of recent papers use a "leave-last-out" protocol, which holds out the final interaction for each user. Because users have different timelines, training data often contains events that occur after test events, allowing models to effectively "learn from the future."
- Lack of Realism: Most benchmarks don't separately measure performance on new users (cold-start) or new items (novelty) in a future time period (out-of-time).
- Data Quality: Many public datasets lack the precise calendar timestamps required to enforce a true chronological split.
Methods
The core of the paper is the INTENT evaluation grid. It categorizes every test event into one of seven cells based on three binary axes:
- User: Is the user known from training, or are they a 10% held-out "new" user?
- Time: Did the event happen before or after a fixed per-dataset cutoff date?
- Item: Had the target item appeared in the training set?
To ensure the benchmark is robust, the authors conducted a massive audit of 59 public dataset families. They enforced a strict "global time split" rule: roughly 80% of all events must precede the cutoff date. This ensures that no training event follows any test event in the "future" cells.
Datasets
After auditing 101 public releases, only five datasets met all criteria (stable entities, calendar timestamps, >25k users, clear action mapping, and accessibility).
The selected "Big Five" are:
- Amazon H&K: Sparse data spanning 25 years.
- KuaiRand: Includes random exposure and explicit negative feedback.
- MicroLens: Rich item content (video, text, audio).
- H&M: Includes static customer profiles and prices.
- REES46: A multi-category "purchase funnel" (view → cart → buy).
Key Insights and Ideas
- "Time costs more than a new user": Models generally perform similarly on known and new users within the same time period. However, crossing the "time cutoff" into the future causes a massive performance drop (25% to 68% for the SASRec model). This suggests that shifts in global behavior over time are a much bigger challenge than individual user cold-starts.
- The Memorization Gap: Standard identifier-based models like SASRec score zero on new items. They have effectively "memorized" the catalog.
- Metadata is not a silver bullet: Adding item metadata (SASRec-F) only significantly helped on the content-rich MicroLens dataset. On others, it barely moved the needle for new items.
- Repeat-Last is a powerful baseline: Simply predicting that a user will repeat their last action is often the best strategy for "new items" (items the model hasn't seen, but the user has interacted with earlier in the test period).
Key Results
The authors evaluated four models: a Popularity baseline, a "Repeat-last" control, the SASRec transformer, and a metadata-aware variant (SASRec-F).
- SASRec dominates on "seen items" in the future, often doubling the performance of popularity-based baselines.
- Repeat-last wins on "new items" in four out of five datasets. This highlights that a user's recent history is often more predictive of a "new-to-model" item than any learned representation.
- On MicroLens, SASRec-F was the only model to achieve a non-zero score on new items for new users, thanks to the rich video metadata.
Limitations and Future Work
- Platform Confounding: Because each unique condition (like "purchase funnel") is tied to a specific dataset, it's hard to tell if a result is due to the condition or the platform itself.
- Untuned Models: The reference models were used with standard hyperparameters; tuning might shift the rankings.
- Public vs. Private Data: No public dataset perfectly mirrors a bank's ledger. The authors found that existing "public banking datasets" often fail the ordering criterion due to anonymization noise.
- Future Directions: The authors suggest that future work should focus on models that can ingest knowledge from outside the training split, such as pretrained content encoders or cross-platform transfer learning.
Summary updated