TL;DR
Recommender systems are highly vulnerable to both malicious "shilling" attacks and natural data noise, yet evaluating their resilience has historically lacked a unified framework. This paper provides a comprehensive survey of recommender system robustness and introduces ShillingREC, an open-source library designed for the standardized, fair evaluation of adversarial attacks and defenses across both explicit and implicit feedback datasets.
What Problem Does This Paper Address?
Recommender systems (RS) are "open" by design—they rely on user feedback to function. This openness makes them easy targets for manipulation. Malicious actors can inject fake profiles, known as shilling attacks, to artificially inflate a product's popularity (push attacks) or diminish a competitor's reputation (nuke attacks). Such manipulation causes massive economic damage, estimated at over $150 billion annually from fake reviews alone.
Beyond intentional attacks, RS models struggle with "natural" robustness issues like data sparsity (too few interactions), data imbalance, and natural noise in implicit feedback (e.g., a user clicking a link by mistake). While many individual studies address these problems, the field lacks a systematic taxonomy and a shared codebase to benchmark how different models hold up under pressure.
Methods
The authors propose a comprehensive taxonomy that splits robustness into two main categories:
- Adversarial Robustness: Focuses on intentional attacks (heuristic, optimization, GAN, and RL-based) and the corresponding defenses (detection, robust algorithms, and hybrid methods).
- Non-Adversarial Robustness: Addresses inherent data quality issues like sparsity and natural noise.
The core contribution is ShillingREC, a Pytorch-based evaluation library. Unlike previous tools that were limited to specific data types, ShillingREC supports both explicit feedback (ratings) and implicit feedback (clicks/interactions). It integrates a wide array of attack models—ranging from simple "Random Attacks" to sophisticated GAN-based methods like AUSH—and recommendation algorithms like LightGCN and SGL.
Datasets
The paper compiles and categorizes a vast range of standard datasets used for robustness research:
- Explicit Feedback: MovieLens (100k to 20M versions), Amazon Review 2014, Yelp, Netflix, Epinions, FilmTrust, Ciao, CiaoDVD, and Book-Crossing.
- Implicit Feedback: Last.fm, Pinterest, Gowalla, and Adressa 2M Compact.
Key Insights and Ideas
- The Power of Unsupervised Signals: One of the most significant findings is that models incorporating unsupervised signals, such as Self-Enhanced Graph Learning (SGL), show much stronger robustness than purely supervised ones. By using contrastive learning to enforce consistency across different "views" of the data, these models make it much harder for fake, injected users to influence the final recommendations.
- Small Datasets are Fragile: Shilling attacks are significantly more effective on smaller datasets. In these environments, user interactions are sparser, meaning the "biased signals" from a fixed number of fake users carry much more relative weight.
- Attack Sophistication: We have moved past simple rule-based attacks. Modern optimization-based and GAN-based attacks generate fake profiles that don't follow fixed rules, making them more realistic and harder to detect than traditional heuristic methods.
Key Results
- Attack Effectiveness: In implicit feedback tests, GAN-based (AUSH) and optimization-based (RevAdv) attacks consistently outperformed traditional heuristic attacks like "Bandwagon" across almost all recommendation models.
- Model Resilience: In head-to-head tests, the SGL model consistently outperformed LightGCN in robustness metrics (like T-NDCG and T-HR) when subjected to AUSH attacks, confirming the defensive value of contrastive learning.
- Library Utility: The authors demonstrated that ShillingREC provides a "fair" evaluation ground. For instance, it showed that some models previously thought to be robust were actually quite vulnerable when tested against more modern, non-heuristic attacks.
Limitations and Future Work
- Simulated vs. Real-World Data: A major hurdle is the lack of real-world datasets containing confirmed fraudulent users. Most current research relies on simulated fake profiles, which may not capture the full complexity of professional fraudsters.
- Efficiency vs. Effectiveness: There is a clear trade-off between how effective a defense is and how fast it runs. Many robust algorithms (like those using contrastive learning) add computational overhead, which can be a dealbreaker for systems requiring rapid, real-time inference.
- Feature Richness: Current simulations mostly use simple (user, item, rating) triplets. Future work needs to incorporate richer features like text reviews and user metadata to better simulate—and defend against—real-world scenarios.
Summary updated