PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review

Hua Yang, Aditya Bhardwaj, Shidong Pan, Bowen XuarXiv 2026 · arXiv

As AI coding agents take on more responsibility for code reviews, a critical question emerges: Can they actually spot the subtle, high-stakes privacy defects that land companies in hot water with regulators?

A new paper, "PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review," introduces a massive new resource to answer that question. The authors present PrivacyPR, the largest dataset of its kind, linking real-world privacy bugs to their developer-accepted fixes and the surrounding review context.

TL;DR

PrivacyPR is a new dataset of 4,131 GitHub pull requests (PRs) designed to benchmark how well AI agents catch privacy defects like GDPR violations. The study finds that while AI agents can detect up to 91% of privacy bugs, their success depends heavily on being explicitly told to look for them—and it comes at the cost of frequent false alarms.

What Problem Does This Paper Address?

Privacy regulations like the GDPR and CCPA are notoriously difficult for developers to operationalize. Privacy defects often slip through the cracks, only to be discovered after release via audits or user complaints.

While automated code review is a promising solution, existing benchmarks primarily focus on functional bugs or security vulnerabilities (like SQL injection). Privacy is a different beast; it often involves non-functional requirements like "data minimization" or "consent enforcement" that are hard to define in code. Until now, researchers lacked a large-scale dataset that connected a concrete privacy defect to its accepted fix and the human discussion that led to it.

Methods

The authors built a sophisticated pipeline to construct PrivacyPR:

  1. Search & Filter: They queried GitHub for merged PRs using 17 privacy-related keywords (e.g., "GDPR," "PII leak," "user consent").
  2. Enrichment: They gathered full metadata, including commits, diffs, and review threads.
  3. LLM Annotation: They used Gemini 2.5 Flash-Lite to label PRs for privacy relevance, fix type (e.g., access control, data masking), and specific GDPR principles.
  4. Human Validation: To ensure quality, two human annotators manually verified a subset of 500 PRs.
  5. Evaluation Harness: They developed a tool that "reverts" a privacy fix to reintroduce the original defect, creating a realistic test case for AI reviewers.
Fig. 2. Pipeline of PrivacyPR: (A) dataset construction; (B) the evaluation harness, which builds the Verified and Small splits and reviews Small (Section III).

Datasets

The authors released several versions of the data:

  • PrivacyPR (Full): 4,131 merged PRs from 1,469 repositories. This includes 1,169 high-confidence privacy fixes and 1,643 non-fix PRs.
  • Verified: A manually checked split of 250 privacy fixes and 250 non-fixes.
  • Small: A 100+100 subset used for the paper's core evaluation.

This dataset significantly improves upon previous efforts by providing traceability—linking the bug, the fix, and the review—and coverage across over a thousand diverse repositories.

Key Insights and Ideas

  • Intent Matters More Than Model Tier: One of the most striking findings is that a "cheap" model (like Gemini 3.5 Flash) using a specific privacy-aware prompt outperformed a top-tier model using a generic review prompt. If you don't ask an agent to look for privacy, it probably won't find it.
  • The Context Tax: Giving agents access to the full repository (rather than just the code diff) boosted detection rates from ~60% to ~90%. However, this came with a 10x increase in cost and a significant jump in false alarms.
  • Privacy Fixes are "Heavy": The data shows that privacy-related PRs are generally larger, involve more comments, and take longer to merge than standard PRs. This suggests that privacy is a high-friction area for developers.
  • LLMs as Judges: The authors found that high-end LLMs (like GPT-4o/GPT-6.1) are remarkably consistent at judging whether a review successfully identified a privacy defect, showing high agreement with human verdicts.

Key Results

The evaluation on the "Small" dataset yielded concrete numbers on AI performance:

  • Detection Rate: Under a generic prompt, agents caught only 42–51% of defects. When explicitly prompted to check for privacy, this jumped to 88–91%.
  • False Alarms: Privacy-specific prompting led agents to flag 11–15% of non-fix PRs as having privacy issues.
  • Severity: Agents were able to identify about 77–82% of the defects as "high severity" when prompted correctly.
Comparison of detection rates and false alarms across different models and prompts

Limitations and Future Work

The authors acknowledge several limitations:

  • Keyword Reliance: The initial search might miss privacy fixes that don't use the specific keywords in their titles or descriptions.
  • Data Contamination: Since the dataset uses public GitHub PRs, there is a risk that the models being tested saw these fixes during their pre-training.
  • Schema Scope: The current labeling focuses heavily on GDPR; future versions could expand to include other frameworks or concepts like "accountability."

PrivacyPR is a major step toward making privacy-aware development a reality. By providing a "ground truth" for what a privacy fix looks like, it allows the community to build and test the next generation of AI-powered safeguards.

The dataset and code are publicly available at Zenodo.

Summary updated

Abstract

Coding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defects to their accepted fixes and review context. We present PrivacyPR, a dataset of 4,131 merged GitHub pull requests (PRs) from 1,469 repositories, created between June 2018 and January 2024. Each PR includes its commits, diffs, review discussion, and LLM-generated labels for privacy relevance, fix type, and GDPR principles. PrivacyPR provides 1,169 high-confidence privacy fixes and 1,643 non-fix PRs, together with Verified, a manually checked split of 250 privacy fixes and 250 non-fix PRs, and Small, a 100 + 100 subset for evaluation. On Verified, both annotators endorsed 72% of the LLM's GDPR-principle assignments. Beyond supporting empirical study of how privacy defects arise and are fixed, PrivacyPR provides a benchmark for privacy-aware code review on real, developer-accepted fixes. Its evaluation harness reverts each fix to reintroduce the defect. On Small, two coding agents detect only 42-51% of defects under a generic review prompt, but 88-91% when explicitly asked to check privacy, while flagging 11-15% of non-fix PRs. These results show that privacy defects can be detected by coding agents, but that detection depends strongly on review intent and comes with false alarms. Data, code, reviews, judge verdicts, and an extensible evaluation harness are publicly available at https://doi.org/10.5281/zenodo.23744765.
BibTeX
@misc{yang2026privacypr,
  author = {Hua Yang and Aditya Bhardwaj and Shidong Pan and Bowen Xu},
  title = {{PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review}},
  year = {2026},
  eprint = {2611.18293},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18293},
}
Figure