Human-Inspired Obfuscation for Model Unlearning: Local and Global Strategies with Hyperbolic Representations

Zekun Wang, Jingjie Zeng, Yingxu Li, Liang Yang, Hongfei LinEMNLP 2025

TL;DR

This paper introduces HyperUnlearn, a novel framework for model unlearning that mimics human forgetting by using "fuzzy" data (locally or globally obfuscated) and hyperbolic representations to better isolate and remove sensitive information from large language models (LLMs) while preserving their overall utility. This approach offers a flexible trade-off between privacy and model performance.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful due to their training on massive datasets. However, these datasets can contain sensitive or private information, raising concerns about potential privacy breaches. Model unlearning aims to selectively remove the influence of specific data points from a trained model without the prohibitive cost of retraining from scratch. Existing unlearning methods often struggle to effectively forget sensitive data while simultaneously maintaining the model's general capabilities and performance. This paper addresses this gap by proposing a human-inspired approach that leverages different types of data obfuscation and a specialized geometric space (hyperbolic) to achieve more precise and stable unlearning.

Methods

HyperUnlearn is a three-stage framework:

  1. Fuzzy Data Generation: Inspired by human memory, the authors create two types of "fuzzy" data to simulate forgetting:
  2. Local Fuzzy Data: Obfuscates specific details (e.g., names, dates) while retaining the general context or structure.
  3. Global Fuzzy Data: Obfuscates entire pieces of information, leading to a more complete loss of specific data. This is achieved using prompt engineering with LLMs like DeepSeek-R1.

  4. Semantic Representation: To improve the precision of unlearning, the "unlearn" data is projected into hyperbolic space using the Lorentz model. Hyperbolic geometry is known for its ability to represent hierarchical and clustered data more compactly than Euclidean space, which helps to tighten the semantic distribution of the data to be forgotten. The "blurred" and "retention" data are represented in Euclidean space.

  5. Freezing-Aware Unlearning: To prevent "spurious unlearning" (where the model might inadvertently reverse forgetting effects or lose general knowledge), the lower layers of the LLM are frozen. This ensures that unlearning operations are confined to the higher layers, which are more sensitive to specific task-oriented patterns, thereby preserving the model's core capabilities. The unlearning process involves a combination of gradient ascent (for forgetting) and gradient descent (for retaining knowledge).

Datasets

The experiments utilize a benchmark dataset specifically designed for machine unlearning tasks, as proposed in Tian et al. (2024). This dataset consists of two subsets: a "Copyrighted Content" dataset and a "User Privacy" dataset. For each subset, the data is further divided into a retain portion and an unlearn portion, with separate splits for training and validation. The authors also use DeepSeek-R1 for generating synthetic fuzzy data.

Key Insights and Ideas

  • Human-inspired forgetting is effective: Mimicking how humans forget (locally or globally) provides a more nuanced approach to unlearning than simply trying to erase data.
  • Hyperbolic space aids precision: Projecting data into hyperbolic space helps to cluster semantically similar information more tightly, enabling more targeted and efficient forgetting.
  • Layer freezing is crucial for stability: Freezing early layers of the model prevents the re-alignment of previously unlearned content, ensuring that forgetting is more permanent and that general capabilities are preserved.
  • Local vs. Global fuzziness offers a trade-off: Local fuzziness is better for retaining model utility and fluency, while global fuzziness leads to stronger forgetting but can degrade generation quality. This allows for flexible application-specific tuning.
  • Standard metrics can be misleading: The paper argues that high perplexity (PPL) on unlearned data doesn't necessarily equate to good unlearning, as it can also indicate degraded model quality. They introduce LLM-based Privacy Check (LPC) as a more reliable metric for assessing actual privacy suppression.

Key Results

  • HyperUnlearn achieves the best performance on the Retention Success metric, indicating strong preservation of general knowledge.
  • While its Unlearn Success scores might appear less competitive than more aggressive methods on certain metrics, the LLM-based Privacy Check (LPC) reveals that HyperUnlearn significantly suppresses critical content leakage, confirming its practical unlearning effectiveness.
  • Experiments on LLaMA2-7B and Qwen2.5-7B models demonstrate that HyperUnlearn effectively removes sensitive information while preserving model utility.
  • The paper shows that models trained with global fuzzy data achieve stronger forgetting but degraded quality, while local fuzzy data preserves generation ability but results in weaker forgetting, highlighting the trade-off.

Limitations and Future Work

  • Computational Constraints: All experiments were conducted on 7B-scale models. The generalizability of the findings to larger models (e.g., 13B, 70B) remains to be validated.
  • Future Work: The authors suggest exploring the scaling of their method to larger language models.

Summary updated

BibTeX
@inproceedings{wang2025human,
  author = {Zekun Wang and Jingjie Zeng and Yingxu Li and Liang Yang and Hongfei Lin},
  title = {{Human-Inspired Obfuscation for Model Unlearning: Local and Global Strategies with Hyperbolic Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure