Towards Robust Recommendation: A Review and an Adversarial Robustness Evaluation Library

Lei Cheng, Xiaowen Huang, Jitao Sang, Jian Yu

Abstract

Recently, recommender systems have achieved significant success. However, due to their openness, they remain vulnerable to malicious attacks and natural noise. This survey provides a comprehensive overview of robustness in recommender systems, categorizing it into adversarial and non-adversarial robustness. We introduce fundamental principles, classical methods, and summarize commonly used datasets and metrics. Furthermore, we propose ShillingREC, an adversarial robustness evaluation library, to facilitate fair and efficient evaluation of attack and defense methods.

Introduction: The Vulnerability of Recommendation

Most recommendation algorithms prioritize accuracy over stability. Because these systems are "open"—allowing users to create accounts and provide feedback—they are prime targets for shilling attacks. In these scenarios, attackers inject fake user profiles to "push" (promote) or "nuke" (demote) specific items. Beyond malicious intent, non-adversarial factors like data sparsity, natural noise in implicit feedback, and data imbalance also degrade performance. The community has lacked a unified way to categorize these threats and a standardized tool to evaluate how well models stand up to them.

A Taxonomy of Robustness

The field of robust recommendation can be broadly divided into two categories: adversarial and non-adversarial robustness.

Taxonomy of Robust Recommendation research, categorizing adversarial and non-adversarial approaches.

1. Adversarial Robustness

This focuses on intentional attacks designed to manipulate system outputs. These are categorized by their underlying mechanisms:

  • Heuristic-based: Simple strategies like "Random" or "Average" attacks.
  • Optimization-based: More sophisticated attacks that use bi-level optimization to generate effective fake data.
  • GAN-based: Using Generative Adversarial Networks to create "stealthy" fake profiles that mimic real user behavior.
  • RL-based: Leveraging Reinforcement Learning to optimize attack sequences.

Defenses against these threats include detection mechanisms, robust training algorithms, and hybrid methods.

2. Non-Adversarial Robustness

This addresses "natural" data issues. Key challenges include handling Data Sparsity, filtering Natural Noise in implicit feedback, and addressing Data Imbalance to ensure the model doesn't over-rely on popular items.

ShillingREC: A Standardized Evaluation Library

To address the lack of unified evaluation tools, the authors introduced ShillingREC. Built on PyTorch, this modular library allows researchers to inject fake data into clean datasets using various attack models and test them against standard models like MF, LightGCN, and SGL.

Summary of various attack methods categorized by type, data feedback, and required knowledge.

ShillingREC is unique in its support for implicit feedback (clicks/views), reflecting how modern industrial recommenders actually operate. It evaluates results using specialized metrics like Prediction Shift (PS) and Targeted Hit Rate (T-HR).

Key Insights and Results

Benchmarking via ShillingREC revealed several critical findings:

  • Fragility of Collaborative Filtering: CF models are inherently vulnerable because they rely on collective user behavior, which attackers can mimic to hijack the system's "intelligence."
  • The Strength of Self-Supervision: Models using unsupervised contrastive learning (like SGL) are significantly more robust. The contrastive tasks act as a filter, helping the model focus on consistent global patterns rather than local "outlier" signals from attackers.
  • Attack Sophistication: GAN-based and Optimization-based attacks are far more destructive and harder to detect than traditional heuristic methods, as they generate profiles that look statistically identical to real users.
  • Scale Matters: Smaller or sparser datasets are much easier to manipulate, as fake signals can more easily drown out real user data.

Future Directions

Despite progress, several challenges remain. Most research relies on simulated fake users; there is a dire need for benchmarks based on actual, real-world fraudulent behavior. Additionally, robust algorithms must be optimized for high-timeliness production environments where efficiency is critical. Finally, developing a "universal" defense strategy that works across different attack types remains a primary goal for the community.

The ShillingREC project is available at: https://github.com/chengleileilei/ShillingREC

Summary updated

Abstract

Recently, recommender system has achieved significant success. However, due to the openness of recommender systems, they remain vulnerable to malicious attacks. Additionally, natural noise in training data and issues such as data sparsity can also degrade the performance of recommender systems. Therefore, enhancing the robustness of recommender systems has become an increasingly important research topic. In this survey, we provide a comprehensive overview of the robustness of recommender systems. Based on our investigation, we categorize the robustness of recommender systems into adversarial robustness and non-adversarial robustness. In the adversarial robustness, we introduce the fundamental principles and classical methods of recommender system adversarial attacks and defenses. In the non-adversarial robustness, we analyze non-adversarial robustness from the perspectives of data sparsity, natural noise, and data imbalance. Additionally, we summarize commonly used datasets and evaluation metrics for evaluating the robustness of recommender systems. Finally, we also discuss the current challenges in the field of recommender system robustness and potential future research directions. Additionally, to facilitate fair and efficient evaluation of attack and defense methods in adversarial robustness, we propose an adversarial robustness evaluation library--ShillingREC, and we conduct evaluations of basic attack models and recommendation models. ShillingREC project is released at https://github.com/chengleileilei/ShillingREC.
BibTeX
@article{cheng2025towards,
  author = {Lei Cheng and Xiaowen Huang and Jitao Sang and Jian Yu},
  title = {{Towards Robust Recommendation: A Review and an Adversarial Robustness Evaluation Library}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2025},
  doi = {10.1109/tkde.2025.3581553},
}
Figure