TL;DR
This paper introduces HANKER, an automated framework that uses knowledge graphs to generate comprehensive audit datasets for evaluating LLM unlearning. HANKER addresses the limitations of existing benchmarks by ensuring thorough coverage and removing redundant information, revealing that current unlearning evaluations may significantly overestimate effectiveness due to this redundancy.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly expected to "unlearn" specific information – for instance, to remove sensitive personal data, comply with copyright, or eliminate biased outputs. Evaluating how well LLMs achieve this unlearning is critical, but current benchmarks are often too small and lack the depth to truly assess effectiveness. Two major challenges exist: 1) Audit Adequacy: Ensuring that the generated test cases comprehensively cover all the knowledge that should have been forgotten. 2) Knowledge Redundancy: When information targeted for forgetting also appears in the data that should be retained. Existing methods often fail to account for this overlap, leading to an inaccurate assessment of unlearning success.
Methods
The authors propose HANKER, a three-stage automated framework:
- Knowledge Graph Construction: HANKER first converts both the "forget" and "retain" corpora into structured knowledge graphs (KGs). This breaks down information into atomic units (entities and relationships), allowing for fine-grained analysis. They use the REBEL-large model for this extraction.
- Redundancy Removal: This is a key innovation. HANKER identifies and removes any knowledge facts (edges in the KG) that are present in both the forget and retain KGs. This ensures that the audit focuses only on knowledge that was supposed to be forgotten and not on information that should have been preserved.
- Question Synthesis: Using the pruned knowledge graph (containing only facts to be forgotten), HANKER guides LLMs (specifically DeepSeek-V3) to generate targeted question-answer pairs. This "dual-input" prompting strategy uses both the knowledge triple and its original text context to ensure questions are factually grounded, specific, and assess the intended knowledge.
Datasets
The authors applied HANKER to the NEWS and BOOKS datasets from the existing MUSE benchmark. HANKER significantly expanded these datasets, generating over 69,000 QA pairs for NEWS and 111,000 for BOOKS, compared to MUSE's original ~100 test cases.
Key Insights and Ideas
- The "Right to be Forgotten" is Harder Than It Looks: Current LLM unlearning evaluations might be giving us a false sense of security. The sheer scale of knowledge in LLMs and the subtle ways information can be retained means that simple testing methods are insufficient.
- Redundancy is a Major Evaluation Skew: The most striking insight is how much knowledge redundancy inflates unlearning effectiveness metrics. If a piece of information is meant to be forgotten but also exists in the "retain" set, it's likely to be preserved anyway, making the unlearning method look more successful than it is.
- Knowledge Graphs are Powerful for Auditing: By breaking down information into structured facts, KGs provide a precise way to identify what needs to be forgotten and, crucially, what needs to be kept. This fine-grained approach is essential for robust evaluation.
Key Results
- Massive Scale-Up: HANKER generated datasets orders of magnitude larger than MUSE (e.g., 142x more memorization cases detected on NEWS using ROUGE).
- Dramatically Increased Detection: HANKER identified significantly more knowledge memorization cases (KMCs) across various unlearning methods and metrics. For example, on the NEWS dataset, it found over 4,600 KMCs using ROUGE compared to MUSE's 33, and over 23,600 using Entailment compared to MUSE's 19.
- Quantified Impact of Redundancy: The study showed that without deduplication, ROUGE scores were artificially inflated by 19.7% to 26.1%, and Entailment scores by 32.4% to 35.2%. This highlights that redundancy leads to substantial false positives.
- High-Quality Audit Data: Manual assessment of HANKER's generated QA pairs showed high accuracy in knowledge fact extraction, question-fact relevance, question clarity, and answer-context consistency.
Limitations and Future Work
- Generalization: The current work extends the MUSE benchmark and uses DeepSeek-V3 for question generation. Future work aims to extend HANKER to other benchmarks and explore different LLMs for question generation.
- Ethical Considerations: The authors emphasize that their work is dedicated to improving unlearning evaluation to help researchers understand its effectiveness, which can mitigate risks related to privacy, security, bias, and copyright.
- Distant Supervision: The redundancy removal strategy uses a "distant supervision" approach, which is an approximation and may not perfectly capture all nuances of overlapping knowledge. Future work could explore more sophisticated methods for identifying semantic equivalence.
Summary updated