"Flex Tape Can't Fix That": Bias and Misinformation in Edited Language Models

Karina Halevy, Anna Sotnikova, Badr AlKhamissi, Syrielle Montariol, Antoine Bosselut

Introduction

Model editing is frequently proposed as a "surgical" alternative to the prohibitive costs of retraining Large Language Models (LLMs). By directly modifying model weights to update specific facts—such as a person’s place of birth or current occupation—researchers aim to keep models current. However, this paper reveals that model editing is often less of a scalpel and more of a blunt instrument. The authors introduce SEESAW-CF, a benchmark designed to expose how editing specific facts inadvertently amplifies social biases and degrades the model's knowledge of marginalized demographic groups, often triggering sexist or xenophobic generations.

The Problem: Beyond Simple Specificity

Traditionally, model editing methods are evaluated on "specificity": ensuring that an edit to "Barack Obama" does not accidentally alter facts about "Joe Biden." The authors argue that this metric is insufficient because it treats all errors as equal. If an edit to a white subject’s biography is successful, but a similar edit to a Black subject’s biography causes the model to lose confidence in their citizenship or generate racist tropes, the editing method is fundamentally biased. This research investigates how weight-editing methods create "factual bleedover" across demographic lines and amplify existing societal prejudices.

The SEESAW-CF Benchmark

To study these effects, the researchers developed SEESAW-CF, a novel benchmark for bias detection in model editing. The dataset includes:

  • 3,516 knowledge edits across five key properties: gender, work, language, birth, and citizenship.
  • 734,000 cloze test prompts to measure confidence shifts.
  • 27,000 open-ended prompts for qualitative analysis.
  • Demographic Metadata: Subjects are tagged by race, geographic origin, and gender using Wikidata, allowing for a disaggregated analysis of how edits affect different groups.

Methodology

The study evaluated three prominent weight-based editing techniques:

  1. Constrained Fine-Tuning (FT): Updating weights with a localized objective.
  2. MEND (Model Editor Networks using Gradient Decomposition): A hypernetwork-based approach that transforms gradients for specific edits.
  3. MEMIT (Mass Editing Memory in Transformers): A method that modifies MLP layers to inject multiple facts simultaneously.

These methods were tested across five models: GPT-J-6B, Llama-2 (base and chat), and Mistral-7b (base and instruct).

Key Insights

Knowledge Entanglement and Demographic Bias

The researchers found that a model’s parametric memory is not a clean database; representations are deeply entangled. For instance, representations of "citizenship" are often tied to "race." Editing a subject's citizenship to a country in Africa or Asia frequently reduced the model's confidence in the citizenship of all Black or East Asian subjects in the dataset.

The "Western Bias" Safety Net

Models demonstrated a significantly higher ability to retain facts about North American and Western European subjects post-edit. This suggests that the massive volume of Western data in pre-training creates "stronger" representations that are more resistant to accidental overwriting compared to those of marginalized groups.

Gender-Career Stereotyping

Editing a subject's gender (e.g., from male to female) frequently caused the model to lose confidence in that subject's field of work, particularly in fields like science or politics. This reflects deep-seated stereotypical associations within the model's weights that are triggered by the editing process.

The Fragility of Instruction Tuning

Surprisingly, "instruct" versions of models (like Mistral-instruct) were often more negatively impacted by these biases post-edit than their base counterparts. This indicates that safety tuning is fragile and can be easily undone or bypassed by direct weight editing.

Qualitative Consequences: Sexism and Xenophobia

Beyond confidence scores, the edits led to measurable qualitative harms. Human annotators found that editing a subject's gender led to a massive spike in sexism (a score of 0.509 compared to nearly 0 pre-edit). Similarly, edits to citizenship—especially moving a subject from a European country to a non-European one—led to increased xenophobia and the injection of irrelevant, often harmful, historical or religious "fluff."

Conclusion and Future Work

The study concludes that current model editing techniques are not yet safe for deployment in contexts where demographic fairness is required. While the benchmark is currently limited to English and binary gender constraints, it provides a critical framework for future research. The authors suggest that future work should move beyond synthetic counterfactuals to more realistic updates, such as changing a person's employer, to see if these bias patterns persist in everyday use cases.

Summary updated

BibTeX
@inproceedings{halevy2024flex,
  author = {Karina Halevy and Anna Sotnikova and Badr AlKhamissi and Syrielle Montariol and Antoine Bosselut},
  title = {{"Flex Tape Can't Fix That": Bias and Misinformation in Edited Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.494},
}
Figure