CVTT: Cross-Validation Through Time

Sergey Kolesnikov, Mikhail AndronovarXiv 2022 · OpenReview

Abstract

The evaluation of recommender systems from a practical perspective is a topic of ongoing discourse within the research community. While many current evaluation methods reduce performance to a single value metric as an easy way to compare models, it relies on the assumption that the methods' performance remains constant over time. In this study, we examine this assumption and propose the Cross-Validation Thought Time (CVTT) technique as a more comprehensive evaluation method, focusing on model performance over time. By utilizing the proposed technique, we conduct an in-depth analysis of the performance of popular RecSys algorithms. Our findings indicate that (1) the performance of the recommenders varies over time for all reviewed datasets, (2) using simple evaluation approaches can lead to a substantial decrease in performance in real-world evaluation scenarios, and (3) excessive data usage can lead to suboptimal results.

Introduction: The Static Evaluation Blind Spot

In Recommender Systems (RS) research, performance is typically reported as a single static metric, such as NDCG@10. This approach assumes that a model’s effectiveness is constant. However, real-world dynamics—shifting user preferences, new item entries, and data obsolescence—challenge this assumption. Traditional "Random Splits" often introduce data leakage by allowing models to "see the future" during training, leading to over-optimistic results that fail in production.

Methodology: Cross-Validation Through Time (CVTT)

To address these issues, we propose CVTT (Cross-Validation Through Time). Instead of a single split, the dataset is divided into regular chronological intervals or "folds."

The CVTT pipeline: iterative hyperparameter optimization, training, and evaluation over chronological folds.

As shown in Figure 1, for every time step, the process follows a strict pipeline:

  1. Hyperparameter Optimization: Settings are optimized using a "train" and "validation" set from the past relative to the fold.
  2. Training: The model is retrained on the combined train+validation data.
  3. Evaluation: The model is tested on the subsequent "future" time interval.
  4. Iteration: This repeats across all folds, generating a performance time series.

We compare two primary data strategies:

  • Expand: The training set grows cumulatively, including all historical data.
  • Window: The training set includes only the $N$ most recent intervals, discarding stale data.

Experimental Setup

We evaluated CVTT using three major benchmarks:

  • Amazon Movies and TV: ~8 million interactions (1996–2018).
  • MovieLens 20M: 20 million ratings.
  • Yelp: ~2 million users' business reviews. Datasets were filtered to ensure continuous user/item activity across periods.

Key Insights: The "More Data" Paradox and Volatility

Our analysis revealed several critical insights:

  • The "More Data" Paradox: Counter-intuitively, using all historical data (Expand) often yields worse results than using a smaller window of recent data. Older interactions can introduce noise that obscures current trends.
  • Performance Volatility: Accuracy is not a flat line; it fluctuates significantly over time. A model superior in one period may be outperformed in the next.
  • Misleading Random Splits: In 75% of experimental pairs, models performed differently on temporal partitions versus random ones. Random splits provide an "easy" test that fails to reflect the difficulty of future prediction.

Key Results

The effectiveness of windowing varies by model and dataset. For instance, with the iALS model, the Window-3 strategy (using the last 3 months) frequently outperformed the Expand strategy on Yelp and MovieLens.

Performance comparison (NDCG@10) of iALS across three datasets using different data strategies: Expand, Random Expand, Window-3, and Random Window-3.

Figure 6 illustrates how the data strategy significantly impacts performance across different datasets. While some models like SLIM show stability, others like ItemKNN are highly sensitive to data updates. Our meta-analysis of over 30 major RS papers confirms that temporal splitting remains underutilized despite being the only strategy to effectively prevent data leakage.

Limitations and Future Work

CVTT is computationally expensive due to iterative retraining and re-optimization. Future work should explore treating the "window length" as a hyperparameter itself, allowing models to dynamically determine the optimal amount of historical context needed. Additionally, further research is required to understand how CVTT handles extreme "cold start" scenarios versus general data drift.

Summary updated

Abstract

The evaluation of recommender systems from a practical perspective is a topic of ongoing discourse within the research community. While many current evaluation methods reduce performance to a single value metric as an easy way to compare models, it relies on the assumption that the methods' performance remains constant over time. In this study, we examine this assumption and propose the Cross-Validation Thought Time (CVTT) technique as a more comprehensive evaluation method, focusing on model performance over time. By utilizing the proposed technique, we conduct an in-depth analysis of the performance of popular RecSys algorithms. Our findings indicate that (1) the performance of the recommenders varies over time for all reviewed datasets, (2) using simple evaluation approaches can lead to a substantial decrease in performance in real-world evaluation scenarios, and (3) excessive data usage can lead to suboptimal results.
BibTeX
@misc{kolesnikov2022cvtt,
  author = {Sergey Kolesnikov and Mikhail Andronov},
  title = {{CVTT: Cross-Validation Through Time}},
  year = {2022},
  eprint = {2205.05393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.05393},
}
Figure