Abstract
We present techniques to characterize which data is important to a recommender system and which is not. Important data is data that contributes most to the accuracy of the recommendation algorithm, while less important data contributes less to the accuracy or even decreases it. Characterizing the importance of data has two potential direct benefits: (1) increased privacy and (2) reduced data management costs, including storage. For privacy, we enable increased recommendation accuracy for comparable privacy levels using existing data obfuscation techniques. For storage, our results indicate that we can achieve large reductions in recommendation data and yet maintain recommendation accuracy. Our main technique is called differential data analysis. The name is inspired by other sorts of differential analysis, such as differential power analysis and differential cryptanalysis, where insight comes through analysis of slightly differing inputs. In differential data analysis we chunk the data and compare results in the presence or absence of each chunk. We present results applying differential data analysis to two datasets and three different kinds of attributes. The first attribute is called user hardship. This is a novel attribute, particularly relevant to location datasets, that indicates how burdensome a data point was to achieve. The second and third attributes are more standard: timestamp and user rating. For user rating, we confirm previous work concerning the increased importance to the recommender of data corresponding to high and low user ratings.
Introduction
Recommender systems have become ubiquitous, relying on massive datasets to predict user preferences. However, the collection of such vast amounts of behavioral data—ranging from location check-ins to movie ratings—introduces significant challenges regarding user privacy and data management costs. This research addresses a fundamental question: Is all this data necessary for accurate recommendations? By identifying "high-signal" data points versus "noise," we can optimize systems for both privacy and efficiency without compromising performance.
Methodology: Differential Data Analysis (DDA)
We introduce Differential Data Analysis (DDA), a technique inspired by differential cryptanalysis. DDA identifies the utility of data subsets through a systematic omission process:
- Chunking: Training data is divided into ten equal deciles based on specific attributes, such as user rating or "User Hardship."
- Omission Testing: The recommendation model is retrained ten times, each time omitting a different decile.
- Impact Measurement: We measure the change in accuracy (e.g., RMSE or Precision@5) resulting from each omission.
- Z-Score Ranking: Chunks are assigned a usefulness score. Chunks whose removal significantly degrades accuracy are deemed "high-signal," while those whose removal improves accuracy are classified as "noise."
A novel metric introduced for location data is User Hardship, which quantifies the effort a user exerted to reach a location, often calculated by distance from their typical activity centers (KMeans centroids).
Experimental Results
Location Data and User Hardship
Applying DDA to the Gowalla dataset across four major cities revealed that data representing a user's routine—their "usual haunts" with low-to-mid hardship—provides the strongest signal. Conversely, data collected during late-night hours (2 AM to 4 AM) often acted as noise. Removing this noise actually improved recommendation accuracy.
Movie Ratings and Signal Extremes
In the MovieLens 1M dataset, DDA confirmed that extreme ratings (1-star and 5-star) carry significantly more signal than neutral 3-star ratings. As shown in the figure below, removing the first or tenth deciles (the extremes) leads to a much sharper increase in RMSE compared to removing middle deciles.
Key Applications
Data Reduction and Storage
On the Gowalla Austin dataset, we successfully removed approximately 40% of the training data—specifically high-hardship deciles and the 2 AM–4 AM window—while maintaining recommendation accuracy comparable to the full dataset. This suggests significant potential for reducing storage costs and legal liabilities.
Intelligent Privacy Suppression
DDA enables "intelligent obfuscation." Instead of random data suppression, users can suppress low-signal data that the recommender does not need. At a 30% deletion rate, this intelligent approach maintained a Precision@5 nearly identical to the full-data baseline, significantly outperforming random suppression.
Conclusion and Future Work
Differential Data Analysis provides a robust framework for identifying data utility in recommender systems. While challenges remain—such as the "bootstrap problem" of needing initial data to calculate importance—the stability of these rankings across different data splits is promising. Future research will explore temporal dynamics and the sensitivity of these findings to diverse recommendation architectures.
Summary updated