Addressing the Generality-Locality Trade-off in Knowledge Editing

This direction is split into other directions: generality-locality trade-off. Its last version stays readable and citable here.

Version 1 of 145 papers47 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are typically "frozen" after training. If a fact changes—such as a new prime minister being elected—the model remains stuck with its outdated training data. Traditional solutions like full retraining are prohibitively expensive, while standard fine-tuning often leads to "catastrophic forgetting," where the model learns the new fact but loses its general reasoning abilities or corrupts unrelated knowledge.

Knowledge Editing (KE) is a suite of techniques designed to perform "brain surgery" on these models, surgically updating specific facts without damaging the rest of the network.

The Impossible Triangle of Model Editing

Researchers have identified a fundamental tension in model editing, often referred to by Peng Wang and colleagues as the Impossible Triangle or the Generality-Locality Trade-off. An ideal edit must satisfy three conflicting criteria:

  1. Reliability: The model must consistently provide the new answer when asked the specific target question.
  2. Generality: The model must understand the meaning of the update, applying it to paraphrased questions (e.g., if "The CEO of X is Y," it should also know "Y leads X").
  3. Locality: The edit must be "local," meaning it should not accidentally change unrelated facts (e.g., updating the CEO of Apple shouldn't change the CEO of Microsoft).

As Eric Mitchell and Kaihang Pan observe, methods that excel at reliability often struggle with locality, inadvertently "leaking" the change into unrelated areas of the model's knowledge base.

Locate and Edit: The Associative Memory View

One of the most influential frameworks for editing is the "locate-then-edit" approach. In ROME, Kevin Meng and David Bau discovered that factual knowledge is often stored in localized "early sites"—specifically the feed-forward networks (MLPs) of middle layers.

These MLPs act like linear associative memories. They store keys (subjects like "The Eiffel Tower") and values (attributes like "is in Paris"). By treating the weight matrix as a key-value store, researchers can use a rank-one update to "overwrite" a specific association. This approach was scaled by Chenmien Tan in the MALMEN framework, which uses meta-learning to aggregate thousands of these parameter shifts simultaneously.

Protecting the Rest: Null-Space and Orthogonality

To solve the locality problem, researchers have turned to linear algebra. If we think of a model's knowledge as a high-dimensional space, an edit is a vector that moves the model's weights.

Jie Shi and Junfeng Fang introduced AlphaEdit, which projects weight updates onto the null space of the model's existing knowledge. By ensuring the update vector is mathematically "invisible" to the weights responsible for unrelated facts, the model can learn new information without distorting its original representations. Similarly, Yuchen Cai developed O-Edit, which ensures each new edit is orthogonal to previous edits, preventing the model from "collapsing" during long sequences of updates.

External Memory: The Sidecar Approach

Rather than modifying the base weights of the model—which is inherently risky—some researchers propose adding "external" knowledge modules.

  • Adapters and LoRA: Lang Yu introduced MELO, which uses a vector database to dynamically activate small, specific LoRA (Low-Rank Adaptation) blocks. When a query matches an edited fact, the model routes the computation through a specialized "plug-in" block.
  • Mixture of Experts: Piji Li and Renzhi Wang developed MEMoE, which uses a bypass structure to route knowledge updates through specific "experts" without altering the original parameters.

These methods, including Ke Wang’s MEMOIR, are particularly effective for lifelong editing, where a model must learn thousands of facts over time without the "interference" that plagues direct weight-tuning.

The Frontier: Reasoning and Ripple Effects

The most significant remaining challenge is portability—the ability of an edit to propagate through the model's reasoning chains. As Qizhou Chen notes in the UNIEDIT benchmark, current editors can easily change "The CEO of X is Y," but the model often fails to realize that "Y now works at X" or that "X's leadership has changed."

Zhenyu Lei and colleagues suggest that this is because reasoning is stored in "neural circuits." Their REdit framework attempts to "reshape" these circuits rather than just swapping facts. By disentangling overlapping circuits, they aim to allow edits that generalize to complex multi-hop reasoning without breaking the model's underlying logic.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_addressing_the_generality_locality_trade_f7c0755d,
  title = {Addressing the Generality-Locality Trade-off in Knowledge Editing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/addressing-the-generality-locality-trade-off-in-knowledge-editing/rdr_f7c0755d8b874c3f9fcc0c45ccbcf2c6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_addressing_the_generality_locality_trade_f7c0755d_v1,
  title = {Addressing the Generality-Locality Trade-off in Knowledge Editing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/addressing-the-generality-locality-trade-off-in-knowledge-editing/rdr_f7c0755d8b874c3f9fcc0c45ccbcf2c6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}