StepKE: Stepwise Knowledge Editing for Multi-Hop Question Answering

Jae‐Wook Lee, Dahyun Jung, Heuiseok Lim2025 · DOI

TL;DR

Large Language Models (LLMs) often struggle to use "edited" facts—new information provided after training—when solving complex, multi-step problems. StepKE (Stepwise Knowledge Editing) solves this by decomposing multi-hop questions into a sequence of single-hop sub-questions and explicitly grounding each step with a mix of updated and pre-existing knowledge retrieved from an external graph.

What Problem Does This Paper Address?

Updating an LLM’s knowledge without expensive retraining is known as Knowledge Editing (KE). While we can successfully "hotfix" a single fact (e.g., "The capital of France is now Marseille"), LLMs frequently fail to propagate that change through a reasoning chain.

If you ask a multi-hop question like, "What is the traditional dish of the city where the 2024 Olympics were held?", the model must first identify the city (Paris) and then find the dish. If an edit changed the host city to Marseille, a standard model might still answer with a Parisian dish because it fails to connect the new "host city" fact with its internal "Marseille" knowledge. This "alignment failure" between edited facts and vast pre-existing background knowledge is the core challenge StepKE tackles.

Methods

StepKE introduces a systematic, four-part pipeline to ensure coherent reasoning:

  1. Question Decomposition: The framework breaks a complex multi-hop question into an ordered sequence of single-hop sub-questions. For example, "Who is the CEO of the company that makes the iPhone?" becomes "Who makes the iPhone?" followed by "Who is the CEO of [Company]?"
  2. Knowledge Augmented Generation (KAG): For every sub-question, StepKE retrieves relevant facts from an external Knowledge Graph (KG). Crucially, this KG integrates both the original world knowledge ($K_{orig}$) and the new edits ($K_{edit}$), allowing the model to see how they intersect.
  3. LTE-based Fine-tuning: The authors use a "Learning to Edit" (LTE) approach to fine-tune the base LLM. This ensures the model is "reliable" (it actually uses the new facts), "local" (it doesn't forget unrelated facts), and "fluent."
  4. Context-aware Prompting: This is the "glue." As the model moves through the reasoning chain, the prompt accumulates the history of previous steps. It explicitly tells the model: "Given that step 1 was answered by A using fact X, now answer step 2..." This prevents the model from drifting back to its outdated internal weights.

Datasets

The researchers evaluated StepKE on three primary benchmarks designed for multi-hop knowledge editing:

  • MQuAKE-3k: A standard benchmark for multi-hop questions involving edited facts.
  • MQuAKE-2002: A refined version of the above where conflicts between new and old knowledge are specifically resolved.
  • MQuAKE-hard: A more difficult set consisting exclusively of 4-hop questions.
  • 2WikiMultihopQA: Used to test the model’s generalizability to general multi-hop reasoning beyond the specific "editing" format.

Key Insights and Ideas

  • The "Interplay" is Key: Most KE methods treat edits as isolated islands. StepKE recognizes that an edit's value lies in how it connects to the model's existing "background" knowledge.
  • Decomposition Mitigates Drift: By forcing the model to answer one hop at a time, StepKE prevents the "hallucination" that often occurs when an LLM tries to jump straight to a final answer using a mix of new and old (conflicting) data.
  • The "E4" Error Type: The authors identified that the most common failure in multi-hop QA is error propagation. If the model makes a tiny mistake in step 1, the entire reasoning chain collapses. StepKE’s stepwise grounding with retrieved facts acts as a "guardrail" against this collapse.
  • Semantic Retrieval Beats Keywords: Using sentence embeddings (semantic similarity) to find facts in the KG proved significantly more effective than traditional keyword matching (BM25), especially when dealing with synonyms or paraphrased questions.

Key Results

StepKE demonstrated a massive performance leap over existing state-of-the-art methods:

  • Accuracy: On MQuAKE-3k (using LLaMA-3.1-8B), StepKE achieved 82.0% accuracy, dwarfing the previous strong baseline, RAE (68.8%), and specialized editing methods like DeepEdit (11.6%).
  • Consistency Across Hops: While other models saw their performance plummet as the number of "hops" increased (e.g., moving from 2-hop to 4-hop questions), StepKE remained remarkably stable, maintaining high accuracy even on the "hard" 4-hop dataset.
  • Generalization: When tested on 2WikiMultihopQA, StepKE outperformed the base model by over 4x (59.8% vs 13.6%), proving that its stepwise approach is a robust strategy for general reasoning, not just a "hack" for edited facts.

Limitations and Future Work

  • Sequential Bias: StepKE is designed for "chain" reasoning. It may struggle with questions requiring parallel reasoning (answering two independent sub-questions and then merging them).
  • Inference Latency: Because the model must generate multiple sub-questions and perform multiple retrievals, it is naturally slower than "end-to-end" models that attempt to answer in one go.
  • Domain Specifics: The authors note that high-stakes fields like law or medicine require even more precise terminology handling than what a general KG retrieval currently provides.
  • Future Direction: Integrating beam search or more advanced re-ranking mechanisms (like cross-encoders) during the retrieval phase could further reduce the "E4" error propagation.

Summary updated

Abstract

Knowledge editing aims to update Large Language Models (LLMs) with new information without costly retraining.However, consistently reflecting these updates in complex multihop Question Answering (QA), which demands reasoning over interconnected facts, is challenging.Many existing methods overlook the interplay with pre-existing knowledge, leading to inconsistent edit propagation.To overcome this, we introduce StepKE (Stepwise Knowledge Editing for Multi-hop QA), a novel framework for robustly integrating edited and existing knowledge for coherent multi-hop reasoning.StepKE uniquely decomposes multihop questions into sequential single-hop subquestions, retrieving relevant facts (both edited and pre-existing) from an external knowledge graph for each step.It employs context-aware prompting with prior reasoning history and finetuning for precise edit propagation.This systematic integration enables effective stepwise reasoning.Experiments show StepKE generates significantly more accurate and consistent responses than baselines, showcasing strong knowledge editing and integration in multi-hop QA.
BibTeX
@misc{lee2025stepke,
  author = {Jae‐Wook Lee and Dahyun Jung and Heuiseok Lim},
  title = {{StepKE: Stepwise Knowledge Editing for Multi-Hop Question Answering}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.409},
}
Figure