Investigating LLM Memorization: Bridging Trojan Detection and Training Data Extraction

Manoj Acharya, Xiao Lin, Susmit JhaSafeGenAi @ NeurIPS 2024 · OpenReview

TL;DR

This paper introduces a novel technique using Mutual Information (MI) to measure how much Large Language Models (LLMs) memorize specific data. This method effectively bridges the gap between understanding memorization and enhancing LLM security by enabling better detection of backdoor (Trojan) attacks and more efficient extraction of training data, outperforming existing baselines on both tasks.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly integrated into critical applications, but their training on vast, often unmonitored datasets from the internet raises significant security concerns. One major threat is "backdoor" or "Trojan" attacks, where attackers embed hidden trigger patterns in the training data. These poisoned models behave normally until a specific trigger is present in the input, at which point they produce malicious or unintended outputs. Detecting these attacks is challenging because they often mimic normal behavior. Furthermore, LLMs are known to memorize parts of their training data, which can lead to privacy risks and the potential for extracting sensitive information. Existing methods for detecting Trojans often rely on assumptions about attack methodologies or trigger patterns, and training data extraction techniques can be inefficient. This paper aims to address these challenges by providing a unified, assumption-free approach to measure memorization, which can then be leveraged for both Trojan detection and training data extraction.

Methods

The core of this work is a novel scoring method based on Mutual Information (MI) to quantify memorization. MI measures how much information one random variable provides about another. In this context, the authors use MI to assess how much information a given input prefix provides about its corresponding output suffix.

The proposed Memorization Score (MS) is derived from the MI formula:

$MS(x,y) = P(x, y) \log \frac{P(x, y)}{P(x)P(y)}$

This score is designed to be sensitive to rare events or sequences that are memorized but not frequently encountered, making it effective for detecting both benign and malicious memorization. For a given input sequence, the MS is calculated as the maximum score across all possible prefix-suffix split points. This approach is presented as an improvement over methods that rely on average log probabilities, as it directly analyzes the dependency of suffixes on prefixes, reflecting a deeper form of memorization. The authors also detail how to compute the necessary probabilities ($P(x)$, $P(y)$, and $P(x,y)$) from the LLM.

Datasets

The paper evaluates its approach on two existing benchmarks:

  • TrojAI challenge dataset (llm-pretrain-apr2024): This dataset focuses on detecting backdoors in LLMs. It includes Llama2-7B models trained on causal language modeling, with some models intentionally poisoned using full fine-tuning or LoRA fine-tuning.
  • lm-extraction-benchmark: This benchmark is designed for training data extraction from LLMs. It uses the GPT-Neo 1.3B model and its memorization of "The Pile" training set. The task involves extracting a correct 50-token suffix given a 50-token prefix.

Key Insights and Ideas

  • Unified approach for memorization: The paper demonstrates that measuring memorization using Mutual Information can serve as a single, powerful metric for both detecting malicious backdoor attacks and extracting training data. This bridges a conceptual gap between these two related LLM security and privacy concerns.
  • Sensitivity to rare patterns: The MI-based Memorization Score (MS) is particularly adept at identifying memorized sequences that are not common, which is crucial for detecting targeted Trojan triggers that are often rare by design.
  • Assumption-free detection: Unlike many existing Trojan detection methods that require knowledge of attack patterns or triggers, the MS approach directly measures memorization, making it more generalizable and robust to various attack strategies.
  • Beyond average log probabilities: The MS method offers a more nuanced understanding of memorization by analyzing prefix-suffix dependencies, rather than just the overall likelihood of a sequence, which is a limitation of simpler log-probability-based methods.

Key Results

  • Trojan Detection: On the TrojAI challenge dataset, the proposed MS approach significantly outperforms the average log-prob based baseline, achieving a much lower cross-entropy (CE) value and a perfect Area Under the ROC Curve (AUC) score of 1.0.
  • Training Data Extraction: On the lm-extraction-benchmark, the MS approach consistently outperforms baselines (logp, zlib, and high-conf) in both precision ($M_p$) and recall ($M_R$) metrics. For example, MS achieved a precision of 50.3 and recall of 77.8, outperforming the best baseline by 0.6 on precision and 0.3 on recall. The authors highlight that MS provides these improvements without requiring extensive hyperparameter tuning.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, calculating MI and MS for every potential prefix-suffix split can be computationally intensive, especially for very long sequences.
  • Hyperparameter Sensitivity: Although the authors claim MS provides consistent improvement without hyperparameter tuning, the "soft threshold" used for Trojan detection (10^-4 nats) is a parameter that might require tuning in different scenarios.
  • Future Work: The paper suggests that their method can be extended to other LLM security and privacy tasks. The authors' work opens avenues for further research into understanding and mitigating memorization-related risks in LLMs.

Summary updated

Abstract

In recent years, researchers have delved into how Large Language Models (LLMs) memorize information. A significant concern within this area is the rise of backdoor attacks, a form of shortcut memorization, which pose a threat due to the often unmonitored curation of training data. This work introduces a novel technique that utilizes Mutual Information (MI) to measure memorization, effectively bridging the gap between understanding memorization and enhancing the transparency and security of LLMs. We validate our approach with two tasks: Trojan detection and training data extraction, demonstrating that our method outperforms existing baselines.
BibTeX
@inproceedings{acharya2024investigating,
  author = {Manoj Acharya and Xiao Lin and Susmit Jha},
  title = {{Investigating LLM Memorization: Bridging Trojan Detection and Training Data Extraction}},
  booktitle = {SafeGenAi @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=l974YvnzU5},
}
Figure