Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization

Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi

Abstract

Text anonymization is the process of removing or obfuscating information from textual data to protect the privacy of individuals. This process inherently involves a complex trade-off between privacy protection and information preservation, where stringent anonymization methods can significantly impact the text's utility for downstream applications. Evaluating the effectiveness of text anonymization proves challenging from both privacy and utility perspectives, as there is no universal benchmark that can comprehensively assess anonymization techniques across diverse, and sometimes contradictory contexts. We present TAU-EVAL, an open-source framework for benchmarking text anonymization methods through the lens of privacy and utility task sensitivity.

The Privacy-Utility Headache

Text anonymization—the act of stripping personally identifiable information (PII) or obfuscating authorship—is a legal necessity under regulations like GDPR and CCPA. However, it is often a zero-sum game: the more "private" you make a text, the less useful it becomes for training or running other AI models.

The core challenge is that current evaluation methods are fragmented. Researchers often rely on generic metrics like ROUGE or BERTScore to see if the anonymized text "looks like" the original. But these metrics are blind to context. A medical report might have a high ROUGE score after anonymization, but if the specific clinical markers needed for a diagnosis were accidentally redacted, the data is useless. There has been no standard way to measure this task-specific utility loss across diverse domains like healthcare, law, and social media.

The TAU-EVAL Framework

To address this, the authors introduce TAU-EVAL, a modular Python framework built on three distinct pillars:

  1. Anonymization Models: It treats anonymizers as black-box functions, supporting everything from traditional rule-based systems (like Microsoft Presidio) to state-of-the-art Large Language Models (LLMs) via the LiteLLM and Hugging Face ecosystems.
  2. Evaluation Metrics: It includes standard "fidelity" metrics (ROUGE, METEOR, SBERT similarity) but adds "reference-less" metrics like Perplexity to judge the fluency of the resulting text.
  3. Target Tasks: This is the framework's "secret sauce." It integrates the tasksource ecosystem, allowing users to test anonymized data against over 600 structured downstream tasks (e.g., sentiment analysis, NLI, or hate speech detection).
Overview of the TAU-EVAL framework, illustrating the integration of anonymization models, evaluation metrics, and target tasks.

The framework calculates $\Delta_T$ (Utility Sensitivity): the performance drop of a task-specific model when fed anonymized data compared to the original "clean" data.

Key Insights and Results

Generic Metrics are Deceptive

A major finding of the study is the lack of correlation between surface-level metrics and actual task performance. For example, a model might have a high BERTScore but still cause a massive drop in a Natural Language Inference (NLI) task. As shown in the correlation data below, you cannot trust ROUGE or SBERT to tell you if your data is still "useful."

Correlation between standard fidelity metrics and downstream task performance, highlighting the inadequacy of surface-level metrics for measuring utility.

The LLM "Safety" Trap

When LLMs are used to anonymize toxic or hateful text, their internal safety filters often kick in. Instead of just hiding the author's identity, they often "clean up" the toxicity itself. This makes the data useless for training safety-moderation tools, as the very features the model needs to learn are removed.

Size and Task Resilience

  • Size Matters (Inversely): In authorship obfuscation, smaller LLMs (like Llama-3.1-8B) actually provide better privacy than their larger counterparts (70B). Smaller models are "noisier"—they make more aggressive modifications to the text, which hides the author better but destroys more utility.
  • Task Resilience Varies: Sentiment analysis (IMDB) is surprisingly robust to anonymization. However, logical reasoning (ANLI) and medical inference (MedNLI) are extremely fragile; even minor redactions can break the logical chain required for the model to reach the right conclusion.

Conclusion and Future Work

While modern LLMs like Gemini Flash and Llama 3.3 achieve up to 99% recall on PII redaction, the utility cost is high—sometimes causing a 42% drop in F1 score on sensitive tasks like hate speech detection.

The authors suggest moving toward task-aware anonymization—developing models that know which parts of a text are vital for a specific task and ensuring those parts are preserved while everything else is scrubbed. Currently, the framework is focused on English, and future work will explore the unique challenges of multilingual anonymization.

Summary updated

Abstract

Text anonymization is the process of removing or obfuscating information from textual data to protect the privacy of individuals.This process inherently involves a complex trade-off between privacy protection and information preservation, where stringent anonymization methods can significantly impact the text's utility for downstream applications.Evaluating the effectiveness of text anonymization proves challenging from both privacy and utility perspectives, as there is no universal benchmark that can comprehensively assess anonymization techniques across diverse, and sometimes contradictory contexts.We present TAU-EVAL, an open-source framework for benchmarking text anonymization methods through the lens of privacy and utility task sensitivity.A Python library 1 , code, documentation and tutorials 2 are publicly available.
BibTeX
@misc{loiseau2025tau,
  author = {Gabriel Loiseau and Damien Sileo and Damien Riquet and Maxime Meyer and Marc Tommasi},
  title = {{Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization}},
  year = {2025},
  eprint = {2506.05979},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05979},
}

Loading related content...

Figure