GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards

Yule Liu, Heyi Zhang, Jinyi Zheng, Zhen Sun, Zifan Peng, Tianshuo Cong, Yilong Yang, Xinlei He, Zhuo Ma

Abstract

Membership inference attacks (MIAs) on large language models (LLMs) pose significant privacy risks across various stages of model training. Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have brought a profound paradigm shift in LLM training, particularly for complex reasoning tasks. However, the on-policy nature of RLVR introduces a unique privacy leakage pattern: since training relies on self-generated responses without fixed ground-truth outputs, membership inference must now determine whether a given prompt (independent of any specific response) is used during fine-tuning. This creates a threat where leakage arises not from answer memorization. To audit this novel privacy risk, we propose Divergence-in-Behavior Attack (DIBA), the first membership inference framework specifically designed for RLVR. DIBA shifts the focus from memorization to behavioral change, leveraging measurable shifts in model behavior across two axes: advantage-side improvement (e.g., correctness gain) and logit-side divergence (e.g., policy drift). Through comprehensive evaluations, we demonstrate that DIBA significantly outperforms existing baselines, achieving around 0.8 AUC and an order-of-magnitude higher [email protected]%FPR. We validate DIBA's superiority across multiple settings—including in-distribution, cross-dataset, cross-algorithm, black-box scenarios, and extensions to vision-language models. Furthermore, our attack remains robust under moderate defensive measures. To the best of our knowledge, this is the first work to systematically analyze privacy vulnerabilities in RLVR, revealing that even in the absence of explicit supervision, training data exposure can be reliably inferred through behavioral traces.

Introduction: The Privacy Challenge in RLVR

Membership Inference Attacks (MIAs) are critical tools for auditing data privacy and detecting benchmark contamination in Large Language Models (LLMs). Traditionally, these attacks rely on memorization: if a model exhibits unusually high fluency or low loss on a specific text, it likely encountered that text during training.

However, the emergence of Reinforcement Learning with Verifiable Rewards (RLVR)—the paradigm behind models like DeepSeek-R1—has rendered traditional MIAs obsolete. In RLVR, models are not trained on "golden" reference answers. Instead, they generate their own responses and receive rule-based feedback (e.g., a "correct/incorrect" reward for a math problem). Because there is no fixed ground truth to memorize, traditional attacks often perform no better than random guessing.

Comparison of MIA signals across different LLM training stages, highlighting the shift from memorization-based attacks in SFT to behavioral divergence-based attacks in RLVR.

Methodology: The Divergence-in-Behavior Attack (DIBA)

To address this gap, we introduce DIBA (Divergence-in-Behavior Attack). Rather than searching for memorized tokens, DIBA quantifies the behavioral shift a model undergoes during RLVR fine-tuning for a given prompt. The attack compares the fine-tuned model against its original "base" version across two primary axes:

  1. Advantage Side (Performance Gain): It measures the "FT Score"—the improvement in the model's ability to generate correct answers—relative to the "Base Score."
  2. Logit Side (Policy Drift): It analyzes changes in token-level probability distributions. This includes K3 Divergence (an unbiased estimator of policy shift), calibrated Likelihood ratios, and Zlib normalization to account for text complexity.
The DIBA framework: responses from both base and fine-tuned models are processed into sample-level features and aggregated into calibrated behavioral markers for membership prediction.

DIBA utilizes a stacking classifier, combining Random Forest and Logistic Regression, to integrate these diverse behavioral signals into a single, robust membership prediction.

Key Insights and Experimental Results

DIBA significantly outperforms all existing MIA baselines in the RLVR setting, demonstrating that privacy leakage in this paradigm manifests as behavioral change.

Performance and Precision

DIBA achieves a high level of accuracy with an AUC of approximately 0.8 across various settings, whereas traditional attacks hover near 0.5. In privacy-sensitive scenarios requiring a low False Positive Rate (FPR), DIBA achieves a True Positive Rate (TPR) an order of magnitude higher than any individual feature.

The "Residual" Sweet Spot

A key finding is that the attack is most effective on "Residual" prompts—those where the model's performance actually changed during training. Prompts that the model could already solve perfectly or was completely unable to solve leak significantly less information because they induced less behavioral shift.

Algorithm vs. Model Size

Interestingly, the choice of RL algorithm has a greater impact on vulnerability than model size. Models trained with DAPO (Direct Alignment Policy Optimization) were found to be more vulnerable than those trained with GRPO, likely due to DAPO's more aggressive learning dynamics. Model size (3B vs 7B) had a comparatively minor effect.

Generalization and Black-Box Viability

The attack remains robust across different model sizes and datasets, and even transfers to Vision-Language Models (VLMs) with an AUC of ~0.7. Furthermore, DIBA maintains non-trivial performance in "grey-box" scenarios where the attacker uses a proxy model to estimate logits.

Limitations and Future Work

Despite its success, DIBA faces challenges in true black-box settings where only text outputs are available. While the attack works on VLMs, visual-specific behavioral markers could further improve precision. Additionally, while DIBA is robust against standard defenses like KL-regularization, aggressive response paraphrasing can reduce its effectiveness, though at a significant computational cost. Future work should focus on developing more efficient, privacy-preserving RLVR training methods.

Summary updated

Abstract

Membership inference attacks (MIAs) on large language models (LLMs) pose significant privacy risks across various stages of model training. Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have brought a profound paradigm shift in LLM training, particularly for complex reasoning tasks. However, the on-policy nature of RLVR introduces a unique privacy leakage pattern: since training relies on self-generated responses without fixed ground-truth outputs, membership inference must now determine whether a given prompt (independent of any specific response) is used during fine-tuning. This creates a threat where leakage arises not from answer memorization. To audit this novel privacy risk, we propose Divergence-in-Behavior Attack (DIBA), the first membership inference framework specifically designed for RLVR. DIBA shifts the focus from memorization to behavioral change, leveraging measurable shifts in model behavior across two axes: advantage-side improvement (e.g., correctness gain) and logit-side divergence (e.g., policy drift). Through comprehensive evaluations, we demonstrate that DIBA significantly outperforms existing baselines, achieving around 0.8 AUC and an order-of-magnitude higher [email protected]%FPR. We validate DIBA's superiority across multiple settings--including in-distribution, cross-dataset, cross-algorithm, black-box scenarios, and extensions to vision-language models. Furthermore, our attack remains robust under moderate defensive measures. To the best of our knowledge, this is the first work to systematically analyze privacy vulnerabilities in RLVR, revealing that even in the absence of explicit supervision, training data exposure can be reliably inferred through behavioral traces.
BibTeX
@misc{liu2025grpo,
  author = {Yule Liu and Heyi Zhang and Jinyi Zheng and Zhen Sun and Zifan Peng and Tianshuo Cong and Yilong Yang and Xinlei He and Zhuo Ma},
  title = {{GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards}},
  year = {2025},
  eprint = {2511.14045},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.14045},
}
Figure