ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs

Manit Baser, Dinil Mon Divakaran, Mohan GurusamyarXiv 2025 · arXiv

TL;DR

While current model-editing techniques can successfully update or erase specific facts in Large Language Models (LLMs), they fail to prevent "indirect knowledge leakage," where the old information can still be deduced via multi-step reasoning. To address this, the authors introduce ThinkEval, a framework that builds model-specific knowledge graphs to systematically detect this leakage, alongside KnowGIC, a new benchmark dataset designed to evaluate multi-step implication chains.

What Problem Does This Paper Address?

When deploying LLMs in sensitive domains like healthcare, legal, or cybersecurity, it is crucial to correct outdated information, mitigate bias, or erase private data. While model-editing techniques (like ROME or MEMIT) can change a targeted fact in isolation, they often leave causally connected facts untouched.

This creates a vulnerability called indirect knowledge leakage: the supposedly "edited-out" information can still be reconstructed by the model through sequential inference. For example, if you edit the model to believe "Harry Potter studied at Ilvermorny" instead of "Hogwarts," but leave his affiliation with Gryffindor house intact, a user can still prompt the model through a chain of related questions to deduce the original, unedited fact.

Figure 1: Example of extracting the original fact post-edit. Prompting with a direct query may fail, but a 3-step sequential inference may extract the original fact.

Methods

The authors propose ThinkEval, a framework that operationalizes a new evaluation setting called deep editing. Instead of evaluating edits using single, isolated queries, deep editing tests whether an edit propagates consistently across a network of logically connected facts.

ThinkEval works by automatically constructing a specialized, model-specific knowledge graph using Chain-of-Thought (CoT) reasoning:

  1. Query Validation and Refinement: It checks if the unedited LLM actually knows a given fact triplet (subject, relation, object).
  2. Automated Triplet Generation: It uses CoT prompting to extract intermediate reasoning pathways from the model, segmenting sentences into atomic triplets.
  3. Graph Synthesis and Chain Sequencing: It organizes these triplets into a directed knowledge graph and generates $n$-step sequential implication chains to test the model post-edit.
The ThinkEval framework pipeline, showing dataset generation from an original model, model editing, and deep editing evaluation.

To quantify performance, the paper introduces a new metric: Indirect Fact Recovery (IFR). IFR measures the probability that the original target fact can still be reconstructed across reasoning paths post-edit. This is paired with a Preservation metric, which evaluates how well the editing technique retains unrelated, broader contextual knowledge.

Datasets

The authors introduce KnowGIC (Knowledge Graph Implication Chains), a benchmark dataset consisting of 1,406 multi-step sequential-prompting samples spanning chain lengths from 1 to 5 steps.

  • Sources: The dataset draws from diverse relational categories in the existing MQuAKE dataset (e.g., country of citizenship, marital relationships, sports affiliations) and includes a highly targeted, deeply embedded causal relationship sample based on the Harry Potter universe to demonstrate complex knowledge transformations.
  • Format: Unlike prior benchmarks that compress multi-hop reasoning into a single complex query, KnowGIC decomposes reasoning into a sequence of linked queries where the answer to one step serves as the premise for the next.

Key Insights and Ideas

  • The Efficacy Paradox: An editing technique can achieve a perfect score on traditional metrics (Efficacy = 1.0), meaning it always gives the new answer to a direct question, while still failing completely at deep editing because the old fact remains accessible via indirect paths.
  • The Leakage vs. Preservation Trade-off: There is a stark divide in current editing paradigms. Locate-and-edit methods like ROME and sequential methods like PRUNE are highly effective at suppressing indirect facts (low IFR), but they suffer from "over-editing," which severely damages surrounding contextual knowledge (low Preservation). Conversely, AlphaEdit and MEMIT preserve surrounding context beautifully but suffer from massive knowledge leakage.
  • The "Rise-then-Fall" IFR Pattern: For several editing techniques, IFR scores increase from 1-step to 3-step chains before declining at 5 steps. This happens because mid-range chains are short enough to maintain strong implication pathways, but longer chains (4–5 steps) are more likely to hit a broken link caused by the edit's unintended ripple effects.

Key Results

The authors evaluated five parameter-modifying techniques (AlphaEdit, RECT, ROME, MEMIT, PRUNE) across three models (Qwen2.5-7B-Instruct, Llama3-8B-Instruct, GPT2-XL).

  • Harry Potter Case Study: When editing Hogwarts $\rightarrow$ Ilvermorny using AlphaEdit on Llama3-8B, the model achieved a perfect 1.000 Efficacy on direct queries. However, it scored an IFR of 0.780, with 80% of the identified reasoning paths still leaking the original fact (Hogwarts).
  • Model-Wide Trade-offs: On Llama3-8B, ROME achieved the lowest overall knowledge leakage (IFR of 0.134) but degraded surrounding knowledge significantly (Preservation of 0.685). Meanwhile, AlphaEdit maintained excellent context (Preservation of 0.890) but leaked heavily (IFR of 0.509).

Limitations and Future Work

  • Human-in-the-Loop Dependency: While ThinkEval automates triplet generation, human oversight is still utilized to prune hallucinations and verify the validity of the knowledge graph triplets. However, the authors note that using an LLM-based judge (like GPT-4o) can reduce human validation effort by 70–75% with only a marginal loss in accuracy.
  • Scope of Knowledge: The framework is currently optimized for factual model edits (triplets). The authors point out that extending ThinkEval to evaluate abstract concepts, procedural reasoning, or conditional ("if-then") relationships remains an open challenge for future work.
  • Call for Holistic Optimizers: The results demonstrate that future model-editing algorithms must move beyond single-fact updates and explicitly optimize the trade-off between minimizing IFR and maximizing Preservation.

Summary updated

Abstract

Robust model-editing techniques are essential for deploying large language models (LLMs) in practical applications, as they enable cost-effective ways to deal with challenges such as privacy breaches, bias mitigation and misinformation spread. For example, an LLM-based healthcare assistance may need to update out-dated or incorrect knowledge to prevent harmful recommendations. However, many editing techniques focus on isolated facts, which critically fail to prevent indirect knowledge leakage -- the unintended reconstruction of edited-out information through persistent causal links and contextual relationships. To assist users in selecting the right editing technique, we develop and present ThinkEval, a framework to systematically quantify indirect knowledge leakage and ripple effects in model-editing. ThinkEval builds and employs specialized knowledge graphs to analyze the causal structure of facts before and after editing. To support this approach, we present KnowGIC, a benchmark dataset comprising multi-step reasoning paths that precisely measure these complex knowledge transformation effects. We evaluate five editing techniques: AlphaEdit, RECT, ROME, MEMIT, and PRUNE across multiple LLMs. Our results show that these techniques struggle to balance indirect fact suppression with the preservation of related knowledge, compromising the contextual integrity of a model's knowledge. Our dataset is available at: https://github.com/manitbaser/KnowGIC.
BibTeX
@misc{baser2025thinkeval,
  author = {Manit Baser and Dinil Mon Divakaran and Mohan Gurusamy},
  title = {{ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs}},
  year = {2025},
  eprint = {2506.01386},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.01386},
}
Figure