Abstract
We study the practical consequences of dataset sampling strategies on the ranking performance of recommendation algorithms. Recommender systems are generally trained and evaluated on samples of larger datasets. Samples are often taken in a naive or ad-hoc fashion: e.g. by sampling a dataset randomly or by selecting users or items with many interactions. As we demonstrate, commonly-used data sampling schemes can have significant consequences on algorithm performance. Following this observation, this paper makes three main contributions: (1) characterizing the effect of sampling on algorithm performance, in terms of algorithm and dataset characteristics; (2) designing SVP-CF, which is a data-specific sampling strategy, that aims to preserve the relative performance of models after sampling; and (3) developing an oracle, Data-Genie, which can suggest the sampling scheme that is most likely to preserve model performance for a given dataset. Detailed experiments show that using Data-Genie, we can discard upto 5x more data than any sampling strategy with the same level of performance.
Introduction
In Collaborative Filtering (CF), larger datasets typically yield better recommendations but come at the cost of increased training time and environmental impact. To accelerate iteration, practitioners frequently sub-sample data. However, sampling is rarely neutral. Common methods—such as random interaction removal or "head-sampling" (retaining only active users)—can fundamentally alter the relative ranking of algorithms. An algorithm that appears superior on a 10% sample may perform poorly on the full dataset. This paper addresses the challenge of discarding data while ensuring that the relative performance of models remains consistent between the sample and the full-scale production data.
Methodology
The authors propose two primary solutions to the sampling problem:
- SVP-CF (Selection-Via-Proxy for CF): This method utilizes a computationally inexpensive "proxy" model (e.g., simple Matrix Factorization) to identify "hard" examples. By prioritizing interactions that the proxy model finds difficult to rank, SVP-CF assumes that maintaining performance on these challenging cases will preserve the overall model rankings.
- Data-genie: A meta-learning "oracle" designed to select the optimal sampling strategy for a specific dataset. It characterizes datasets using handcrafted features or unsupervised Graph Convolutional Network (GCN) embeddings. A predictive model then identifies which of 16 sampling strategies will best preserve original algorithm rankings.
The study evaluates these methods across six diverse datasets (Amazon Reviews, MovieLens-100k, BeerAdvocate, and Goodreads Comics) covering explicit, implicit, and sequential feedback scenarios.
Results and Discussion
Experimental results reveal several critical insights. Strategies that discard the "tail" (e.g., Head-User sampling) are consistently the least effective at preserving algorithm rankings. Conversely, sampling difficult interactions via SVP-CF is significantly more effective than random sampling. The authors also observed that aggressive sampling often causes simpler models to rise in rankings while complex, parameter-heavy models drop, likely due to overfitting on reduced data.
Data-genie enables researchers to discard up to 5x more data than standard methods while maintaining confidence in benchmarks. Specifically, using only 10% of the data selected via Data-genie resulted in a 5.8x training speedup without compromising the ability to predict the winning algorithm on the full dataset. This efficiency has significant environmental implications, potentially reducing the carbon footprint of a typical development cycle by over 80%.
Conclusion
This work highlights the non-neutrality of dataset sampling in recommendation systems and provides tools to mitigate its distortive effects. While the study covers seven diverse algorithms, future research should expand to industry-scale datasets and investigate the impact of sampling on fairness and demographic bias. Ensuring that "hard" example selection does not inadvertently disadvantage minority groups remains a vital area for future exploration.
Summary updated