TL;DR
SCEN is a novel two-stage training paradigm that allows Large Language Models (LLMs) to be updated with new or corrected information without retraining the entire model. By training individual, lightweight "expert" networks for each piece of knowledge and using dedicated "indexing neurons" to trigger them, SCEN achieves state-of-the-art performance in balancing reliability, generalization to paraphrased queries, and the preservation of unrelated knowledge.
What Problem Does This Paper Address?
Large Language Models (LLMs) are often "frozen in time" based on their training data, leading to factual hallucinations and outdated responses (e.g., naming a former president as the current one). While fine-tuning can update this knowledge, it is computationally expensive and often leads to catastrophic forgetting, where the model loses previously learned information.
The field of Model Editing seeks to fix specific errors efficiently. A successful editing method must satisfy three criteria:
- Reliability: Does the model now give the correct answer for the edited fact?
- Generality: Does it still give the correct answer if the question is rephrased?
- Locality: Does the edit leave unrelated knowledge (e.g., the capital of France) untouched?
Existing methods often struggle with the trade-off between these three. For instance, direct fine-tuning has high reliability but poor locality, while memory-based methods can be hard to scale.
Methods
The authors introduce SCEN (Scalable Model Editing via Customized Expert Networks), a two-stage continuous training paradigm inspired by Mixture-of-Experts (MoE) architectures.
- Stage 1: Training Experts. For every new piece of knowledge to be edited, SCEN trains a lightweight "expert" network. Specifically, it tunes the
down_projweights ($W_{down}$) of a single Feed-Forward Network (FFN) layer for that specific sample while freezing all other model parameters. - Stage 2: Training Indexing Neurons. To ensure the right expert is used at the right time, the authors add a "Scalable Indexing Neuron" for each expert. This neuron is trained using a specialized loss function (including margin loss) to activate only when the model encounters the specific edit sample or its semantically equivalent rewrites.
During inference, the model calculates activation values for these indexing neurons. If an activation exceeds a predefined threshold, the model swaps its original weights for the corresponding "expert" weights to generate the response.
Datasets
The researchers evaluated SCEN using two primary benchmarks on Llama2-7B and 13B:
- ZsRE (Zero-Shot Relation Extraction): A comprehensive question-answering dataset used to measure reliability, generality, and locality.
- Hallucination Dataset: A dataset derived from WikiBio (biographies generated by GPT-3) designed to test a model's ability to mitigate factual hallucinations in long-form text.
Key Insights and Ideas
- Granularity Matters: The authors found that assigning one expert per sample is significantly more robust than trying to force one expert to handle multiple diverse samples. When they tried "compressing" two or four samples into one neuron, performance dropped sharply.
- Knowledge Location: By testing edits across different layers, the study confirms that factual knowledge is predominantly stored in the latter layers of the transformer architecture. Editing the 16th layer and beyond in Llama2-7B yielded much higher reliability than editing earlier layers.
- The Threshold Trade-off: The activation threshold ($\theta$) is a critical knob. Lower values favor reliability and generality but risk damaging locality, while higher values protect original knowledge but might fail to trigger the edit.
Key Results
SCEN demonstrated superior performance across both Llama2-7B and 13B models, particularly in long-sequence editing (up to 1,000 sequential edits).
- ZsRE Performance: On Llama2-7B with 1,000 edits, SCEN achieved an average score of 86.7% across all metrics, significantly outperforming the previous SOTA, GRACE (74.5%), and standard fine-tuning (72.7%).
- Reliability: SCEN reached near-perfect reliability (up to 100% in some 200-edit tests), ensuring that the specific error was indeed corrected.
- Locality: Unlike standard fine-tuning, which saw locality drop to ~38%, SCEN maintained locality scores above 83%, proving it doesn't "break" the rest of the model.
Limitations and Future Work
While SCEN is highly effective, the authors acknowledge that adding a new neuron and expert for every single edit could eventually lead to increased storage requirements and slower inference as the number of edits grows into the thousands or millions.
Future Directions: The team plans to explore advanced weight compression techniques to reduce the footprint of the expert networks and indexing neurons, aiming to make the system even more practical for massive-scale, real-time knowledge updates in production environments.
Summary updated