TL;DR
DynEdit is a new framework for updating factual knowledge in Large Language Models (LLMs) that moves beyond the "one-size-fits-all" approach of previous editors. By using token-level entropy as a guide, it dynamically allocates more editing resources to difficult, information-dense regions of a sequence, resulting in significantly more accurate and robust long-form knowledge updates.
What Problem Does This Paper Address?
Large Language Models inevitably store outdated or incorrect facts. "Knowledge editing" (KE) aims to fix these specific errors by directly modifying model parameters without the massive cost of full retraining.
However, current parameter-editing methods face a "single-token bottleneck." They often focus on a single position (like the last token of a subject) to inject a new fact. While this works for simple triplets (e.g., "Paris is the capital of France"), it fails for long-form knowledge like biographies or code snippets. Recent attempts to solve this using sequential sliding windows treat every token equally. This is inefficient: the model spends as much "editing effort" on predictable function words (like "the" or "is") as it does on critical factual details (like dates or specific names). This "difficulty-agnostic" allocation leads to updates that might look fluent but miss the crucial details they were meant to fix.
Methods
DynEdit introduces a dynamic, entropy-guided approach to sequential editing. It is built on two core components:
- Entropy-Guided Non-Uniform Sliding: Before editing, the system calculates the Shannon entropy for every token in the target sequence using the pre-edit model. High entropy indicates regions where the model is uncertain—typically where the new factual content resides. DynEdit uses this signal to adaptively control its sliding window stride. In high-entropy regions, it reduces the stride to increase window overlap, ensuring these difficult segments are covered by more editing steps.
- Entropy-Guided Token-Level Optimization: Within each editing window, DynEdit doesn't treat all tokens as equal. It assigns higher optimization weights to high-entropy tokens. This ensures that entities, dates, and locations receive stronger supervision than low-information function words, preventing the critical factual signal from being "diluted" during the gradient update.
The method is designed to be "plug-and-play," meaning it can be implemented on top of existing backbone editors like MEMIT or UnKE.
Datasets
The authors evaluated DynEdit across four diverse benchmarks:
- UnKEBench: A dataset of 1,000 long-form counterfactual texts (multi-sentence answers).
- CounterFact: A standard benchmark for structured triplet editing (subject, relation, object).
- MQuAKE: A multi-hop editing dataset that tests if edited facts correctly propagate through reasoning chains.
- EditEverything: A recent benchmark covering diverse formats including mathematics, programming, poetry, news, and chemistry.
Key Insights and Ideas
- Entropy as a Proxy for Importance: The central "aha" moment is the discovery that token entropy is a highly reliable proxy for both editing difficulty and semantic importance. As shown in the paper's analysis, high-entropy tokens are consistently harder for models to rewrite correctly and are much more likely to be content-bearing words (nouns, verbs, entities) rather than function words.
- Solving Gradient Dilution: In long sequences, critical factual tokens are sparse. Uniformly averaging gradients across a window allows "background" tokens to drown out the signal of the important ones. By weighting the loss based on entropy, DynEdit "rescues" the critical update direction.
- Mitigating Error Propagation: By focusing density on difficult regions, DynEdit achieves a much flatter performance drop as sequence length increases compared to baseline methods, which tend to accumulate errors toward the end of long sequences.
Key Results
DynEdit consistently outperformed competitive baselines across multiple models (Llama-3-8B, Qwen-2.5-7B, and Qwen-3-8B):
- Accuracy Gains: It achieved gains of up to +14.0 BLEU points on challenging paraphrase queries compared to AnyEdit.
- Format Robustness: On the EditEverything benchmark, it showed massive improvements in structurally complex domains, such as a +27.03 Rouge-L gain in the mathematics category.
- General Ability: Importantly, the method preserves the model's general capabilities, maintaining comparable scores on benchmarks like MMLU and IFEval while improving specific factual accuracy.
- Efficiency: While it introduces a moderate runtime overhead (roughly 1.5x the time of AnyEdit), it maintains the same peak memory footprint, making it practical for real-world use.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Entropy Limitations: While effective, entropy might not perfectly capture "importance" in every domain, particularly in highly specialized or technical generation tasks.
- Scope: The current framework is optimized for sequential text. Future work could extend these principles to multi-modal editing (images/text) or multi-turn interactive editing.
- Efficiency: Further optimizing the scheduling and weighting operations to reduce the runtime overhead compared to fixed-window methods remains a goal.
Summary updated