Erasing Abstract Concepts from Large Models

16 concepts16 papers1 proposal

Traditional machine unlearning treats a neural network like a database. If a user exercises their "right to be forgotten," or if a training sample contains copyrighted material, the unlearning algorithm attempts to locate and excise that specific data point or token sequence.

However, modern foundation models do not store information in discrete, isolated slots. Instead, they construct highly distributed, abstract semantic representations. If you want a model to forget a person's identity, a dangerous capability (like bioweapon synthesis), or a protected artistic style, simply deleting individual training instances is insufficient. As noted by Xu et al. and Mora et al., instance-level deletion fails to address the broader conceptual knowledge distributed across the model's parameter space. Furthermore, identifying every single training document or image related to a broad concept within a massive, web-scale dataset is practically impossible (Yamashita et al.).

To solve this, a new paradigm has emerged: concept-level unlearning. Instead of tracing data, this approach traces and neutralizes abstract knowledge directly within the model's internal representations.


From Data-Tracing to Knowledge-Tracing

The core intuition behind concept-level unlearning is a shift from data-centric deletion to instruction-centric suppression. Tan et al. formalize this transition by proposing a framework that translates high-level knowledge requests into actionable unlearning objectives. Instead of requiring the original training samples, the authors construct "enhanced forgetting sets" (synthetic prompts or knowledge graphs that map out the target concept) and "coarse-grained retention sets" (to protect unrelated general knowledge).

Data-Centric Unlearning (Traditional):
[Specific Training Sample] ──> Compute Gradients ──> Erase Exact Tokens/Pixels

Concept-Centric Unlearning (Modern):
[Abstract Concept Request] ──> Map Semantic Domain ──> Realign Internal Weights

Why does this work? Because deep learning models organize information based on semantic manifolds rather than exact surface-level wording. Wang et al. demonstrated this by showing that concept-level unlearning is surprisingly resilient to "noisy" forget sets—such as paraphrased text, incomplete snippets, or watermarked content. Because the optimization objectives target the underlying semantic structure rather than token-level saliency, the model naturally generalizes the forgetting process across the entire conceptual domain.


The Mechanics of Conceptual Erasure

To surgically remove a concept without breaking the rest of the model, researchers leverage the model's own internal representations against itself.

In the Erasure of Language Memory (ELM) framework introduced by Gandikota et al., the model is effectively turned into its own critic. By defining a mathematical objective for broad semantic concepts rather than static training sequences, the framework updates the weights so that the model's internal probability distribution for the target concept is flattened or redirected.

For generative models, this redirection can be framed as a geometric steering problem. Simone et al. utilize energy-based proxies within a neural flow matching framework to "softly subtract" undesired semantic regions from a probability distribution. Instead of hard-deleting parameters, this mechanism creates a repulsive force field around the forbidden concept, steering the model's generation pathways away from it while leaving adjacent knowledge intact.


Minimizing Collateral Damage

The primary challenge of concept-level unlearning is avoiding "over-forgetting"—a phenomenon where a model becomes overly defensive or loses general reasoning capabilities after safety training. For instance, if a model is forced to unlearn concepts related to cyberattacks, it might inadvertently refuse harmless programming questions.

To decouple target concepts from benign capabilities, Chen et al. introduced the Prompt Decouple Loss for multimodal models. This loss function explicitly separates the representations of harmful behaviors from the underlying semantic tokens, ensuring that the model only forgets the malicious execution of a concept while retaining its broad utility. Similarly, Huang et al. use a meta-learning framework called Learning-to-Unlearn (LTU), which simulates unlearning steps during training to harmonize conflicting gradients, ensuring that erasing a semantic class does not degrade performance on the rest of the dataset.


The Geometry of Forgetting

Why do some model architectures forget concepts more cleanly than others? The answer often lies in the geometry of their embedding spaces.

Euclidean Space (Flat):
Concepts are entangled across coordinates. Erasing a concept requires 
shifting wide regions, causing collateral damage to nearby points.

Hyperbolic Space (Hierarchical Tree):
Concepts are organized in branches. Excising a high-level concept 
simply means cutting a specific branch, leaving the rest of the tree untouched.

Vidal et al. investigated this by comparing Euclidean and Hyperbolic embedding spaces in contrastive models. Hyperbolic geometry naturally reflects hierarchical structures (like taxonomies or semantic trees). The authors discovered that hyperbolic models are significantly more effective at concept-level erasure because they fundamentally reorganize their internal semantic hierarchies during the unlearning process. Instead of merely breaking local associations, they can cleanly excise an entire high-level branch of knowledge without disturbing the surrounding conceptual architecture.

Key Foundational Work

Machine Unlearning: Challenges in Data Quality and Access (2024) — contributed 2 concept elements to this direction.
KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment (2023) — contributed 2 concept elements to this direction.
Concept Unlearning for Large Language Models (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_concept_level_unlearning_fce39576,
  title = {concept-level unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/concept-level-unlearning/rdr_fce39576dbe743b19898a1df0fa837a7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_concept_level_unlearning_fce39576_v1,
  title = {concept-level unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/concept-level-unlearning/rdr_fce39576dbe743b19898a1df0fa837a7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake