Abstract
Membership Inference (MI) attacks determine whether a specific data point was used during a model's training. While well-established in classification tasks, MI remains challenging for Sequence-to-Sequence (Seq2Seq) models due to their vast output spaces and limited access to internal probability distributions. This paper investigates the vulnerability of text summarization models to MI attacks, introducing a novel "document-only" attack that leverages the robustness of model outputs under input perturbation. Our findings demonstrate that training samples exhibit a "stubbornness" that leaks membership information, even in black-box scenarios.
1. Introduction
As Large Language Models (LLMs) are increasingly integrated into sensitive domains like healthcare and legal services, understanding their privacy risks is paramount. Membership Inference (MI) serves as a fundamental metric for privacy, testing if an adversary can distinguish between training data and unseen samples. In the context of summarization, this risk is particularly acute: if a model's output reveals that a specific medical report or private conversation was part of its training set, the privacy of the individuals involved is compromised.
Historically, MI attacks have relied on fine-grained token-level probabilities. However, many production-grade summarization models are accessed via APIs that only provide the final text. This study addresses whether these "black-box" models are truly resilient to privacy leaks.
2. Methodology
The researchers developed a hierarchy of MI attacks based on the resources available to an attacker, moving from high-resource scenarios to more realistic, low-resource settings.
2.1 Naive Baseline (Reference-Based)
This scenario assumes the attacker possesses both the source document and the original "gold standard" reference summary. By comparing the model's generated summary to the reference using similarity metrics like ROUGE or BERTScore, the attacker can identify training samples. Models tend to memorize and replicate target sequences they encountered during training, leading to higher similarity scores for training data.
2.2 Document Augmentation
This approach utilizes the "max-margin" principle, which suggests that training data points are located further from the model's decision boundaries than unseen data. By perturbing the input document—through sentence swapping or synonym replacement—the attacker measures the stability of the output. Training samples typically yield highly consistent (robust) summaries despite these perturbations.
2.3 Document-Only Attack
The most significant contribution of this work is an attack requiring only the input document. Here, the attacker uses the model’s own initial output as a "pseudo-reference." They then perturb the input and measure how much the subsequent summaries deviate from this pseudo-reference. High consistency across variations serves as a strong signal of membership, making this a potent tool for black-box auditing.
3. Experimental Evaluation
The attacks were evaluated across three distinct datasets to ensure the findings were not task-specific:
- SAMsum: A collection of messenger-like dialogue summaries.
- CNN/DailyMail (CNNDM): A standard benchmark of news articles and highlights.
- MIMIC-cxr: A high-stakes medical dataset containing radiology reports. This is particularly important for privacy research due to the sensitive nature of clinical data.
4. Key Findings
4.1 The Robustness Signal
The study reveals that training data is "stubborn." Because the model was optimized specifically on these training points, it has a much harder time "letting go" of its learned summary even when the input is slightly modified. This robustness is a measurable byproduct of the training process and serves as a reliable indicator for MI.
4.2 Limitations of Beam Search
A key technical insight is that standard confidence scores are often misleading in Seq2Seq models. The beam search process used during decoding naturally favors high-probability sequences, making both seen and unseen data appear "high-confidence." This renders traditional probability-based MI attacks less effective than the proposed robustness-based methods.
4.3 Attack Performance
On the SAMsum and MIMIC datasets, the baseline attack achieved an Area Under the Curve (AUC) exceeding 65% (where 50% represents a random guess). More importantly, the researchers looked at the True Positive Rate at a very low False Positive Rate ([email protected]%FPR). In the MIMIC dataset, the attack successfully identified 305 training samples with only 10 false positives, demonstrating that even if average accuracy is moderate, specific samples can be identified with near-certainty.
5. Defense and Trade-offs
The authors evaluated Differential Privacy (DP-SGD) as a defense mechanism. While DP-SGD successfully reduced the MI attack AUC to near-random levels (50%), it incurred a significant "utility tax." ROUGE scores for the summarization task dropped by approximately 10 points, highlighting the difficult balance researchers must strike between model performance and rigorous privacy guarantees.
6. Conclusion
This research demonstrates that summarization models are not inherently private due to their complex output spaces. The "document-only" attack proves that membership can be inferred even with minimal information. While MI is a critical diagnostic tool, the authors suggest that future work should explore "Extraction" attacks—where an adversary attempts to reconstruct the actual training data—to fully map the privacy landscape of generative AI.
Summary updated