Towards Robust Recommendation: A Review and an Adversarial Robustness Evaluation Library

Lei Cheng, Xiaowen Huang, Jitao Sang, Jian Yu

TL;DR

Recommender systems are highly vulnerable to both malicious "shilling" attacks and natural data noise, yet evaluating their resilience has historically lacked a unified framework. This paper provides a comprehensive survey of recommender system robustness and introduces ShillingREC, an open-source library designed for the standardized, fair evaluation of adversarial attacks and defenses across both explicit and implicit feedback datasets.

What Problem Does This Paper Address?

Recommender systems (RS) are "open" by design—they rely on user feedback to function. This openness makes them easy targets for manipulation. Malicious actors can inject fake profiles, known as shilling attacks, to artificially inflate a product's popularity (push attacks) or diminish a competitor's reputation (nuke attacks). Such manipulation causes massive economic damage, estimated at over $150 billion annually from fake reviews alone.

Beyond intentional attacks, RS models struggle with "natural" robustness issues like data sparsity (too few interactions), data imbalance, and natural noise in implicit feedback (e.g., a user clicking a link by mistake). While many individual studies address these problems, the field lacks a systematic taxonomy and a shared codebase to benchmark how different models hold up under pressure.

Methods

The authors propose a comprehensive taxonomy that splits robustness into two main categories:

  1. Adversarial Robustness: Focuses on intentional attacks (heuristic, optimization, GAN, and RL-based) and the corresponding defenses (detection, robust algorithms, and hybrid methods).
  2. Non-Adversarial Robustness: Addresses inherent data quality issues like sparsity and natural noise.
Fig. 1. Recommender system robustness taxonomy diagram.

The core contribution is ShillingREC, a Pytorch-based evaluation library. Unlike previous tools that were limited to specific data types, ShillingREC supports both explicit feedback (ratings) and implicit feedback (clicks/interactions). It integrates a wide array of attack models—ranging from simple "Random Attacks" to sophisticated GAN-based methods like AUSH—and recommendation algorithms like LightGCN and SGL.

Datasets

The paper compiles and categorizes a vast range of standard datasets used for robustness research:

  • Explicit Feedback: MovieLens (100k to 20M versions), Amazon Review 2014, Yelp, Netflix, Epinions, FilmTrust, Ciao, CiaoDVD, and Book-Crossing.
  • Implicit Feedback: Last.fm, Pinterest, Gowalla, and Adressa 2M Compact.

Key Insights and Ideas

  • The Power of Unsupervised Signals: One of the most significant findings is that models incorporating unsupervised signals, such as Self-Enhanced Graph Learning (SGL), show much stronger robustness than purely supervised ones. By using contrastive learning to enforce consistency across different "views" of the data, these models make it much harder for fake, injected users to influence the final recommendations.
  • Small Datasets are Fragile: Shilling attacks are significantly more effective on smaller datasets. In these environments, user interactions are sparser, meaning the "biased signals" from a fixed number of fake users carry much more relative weight.
  • Attack Sophistication: We have moved past simple rule-based attacks. Modern optimization-based and GAN-based attacks generate fake profiles that don't follow fixed rules, making them more realistic and harder to detect than traditional heuristic methods.
Fig. 2. Diagram of Shilling attack. Blue arrows indicate the recommendation process unaffected by adversarial attacks, while red arrows represent the recommendation process in the presence of adversarial attacks.

Key Results

  • Attack Effectiveness: In implicit feedback tests, GAN-based (AUSH) and optimization-based (RevAdv) attacks consistently outperformed traditional heuristic attacks like "Bandwagon" across almost all recommendation models.
  • Model Resilience: In head-to-head tests, the SGL model consistently outperformed LightGCN in robustness metrics (like T-NDCG and T-HR) when subjected to AUSH attacks, confirming the defensive value of contrastive learning.
  • Library Utility: The authors demonstrated that ShillingREC provides a "fair" evaluation ground. For instance, it showed that some models previously thought to be robust were actually quite vulnerable when tested against more modern, non-heuristic attacks.
Statistics of Shilling Attack Approaches

Limitations and Future Work

  • Simulated vs. Real-World Data: A major hurdle is the lack of real-world datasets containing confirmed fraudulent users. Most current research relies on simulated fake profiles, which may not capture the full complexity of professional fraudsters.
  • Efficiency vs. Effectiveness: There is a clear trade-off between how effective a defense is and how fast it runs. Many robust algorithms (like those using contrastive learning) add computational overhead, which can be a dealbreaker for systems requiring rapid, real-time inference.
  • Feature Richness: Current simulations mostly use simple (user, item, rating) triplets. Future work needs to incorporate richer features like text reviews and user metadata to better simulate—and defend against—real-world scenarios.

Summary updated

Abstract

Recently, recommender system has achieved significant success. However, due to the openness of recommender systems, they remain vulnerable to malicious attacks. Additionally, natural noise in training data and issues such as data sparsity can also degrade the performance of recommender systems. Therefore, enhancing the robustness of recommender systems has become an increasingly important research topic. In this survey, we provide a comprehensive overview of the robustness of recommender systems. Based on our investigation, we categorize the robustness of recommender systems into adversarial robustness and non-adversarial robustness. In the adversarial robustness, we introduce the fundamental principles and classical methods of recommender system adversarial attacks and defenses. In the non-adversarial robustness, we analyze non-adversarial robustness from the perspectives of data sparsity, natural noise, and data imbalance. Additionally, we summarize commonly used datasets and evaluation metrics for evaluating the robustness of recommender systems. Finally, we also discuss the current challenges in the field of recommender system robustness and potential future research directions. Additionally, to facilitate fair and efficient evaluation of attack and defense methods in adversarial robustness, we propose an adversarial robustness evaluation library--ShillingREC, and we conduct evaluations of basic attack models and recommendation models. ShillingREC project is released at https://github.com/chengleileilei/ShillingREC.
BibTeX
@article{cheng2025towards,
  author = {Lei Cheng and Xiaowen Huang and Jitao Sang and Jian Yu},
  title = {{Towards Robust Recommendation: A Review and an Adversarial Robustness Evaluation Library}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2025},
  doi = {10.1109/tkde.2025.3581553},
}
Figure