Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization

Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi

Introduction

Text anonymization—the process of stripping personally identifiable information (PII) or obfuscating authorship—is a legal and ethical necessity in the age of large-scale data processing. However, it creates a fundamental tension: the more "private" a text becomes, the less "useful" it often is for downstream machine learning tasks.

Current evaluation methods for anonymization typically rely on reference-based metrics like ROUGE, METEOR, or BERTScore. While these measure surface-level text similarity, they fail to account for context-dependent utility. For instance, an anonymized medical report is useless if the clinical patterns required for diagnosis are scrubbed along with the patient's name. TAU-EVAL addresses this gap by providing a unified, reproducible framework to measure task-specific degradation across diverse domains.

The TAU-EVAL Framework

TAU-EVAL is an open-source modular Python framework designed to benchmark the trade-off between privacy and utility. The framework is built on three distinct pillars:

  1. Anonymization Models: The framework treats anonymizers as black-box functions, supporting traditional NER-based tools (e.g., Microsoft Presidio) and state-of-the-art Large Language Models (LLMs) like Llama 3.3 and Gemini 1.5.
  2. Evaluation Metrics: It includes both generic fidelity metrics (measuring text similarity) and reference-less metrics such as CoLA for grammatical correctness.
  3. Target Tasks: By leveraging the tasksource ecosystem, TAU-EVAL provides access to over 600 structured tasks for evaluation.
Overview of the TAU-EVAL framework components: Models, Metrics, and Tasks.

The framework formalizes Utility Loss (ΔT) as the difference between a model's performance on original data ($U_{orig}$) versus its performance on anonymized data ($U_{priv}$). This quantification allows researchers to identify exactly which features are lost during the anonymization process.

Experimental Setup

The framework was benchmarked using a suite of datasets covering two primary privacy objectives and eight utility tasks:

  • Privacy Objectives: PII-masking (synthetic dataset with 63 PII classes) and authorship obfuscation (using IMDB62 to hide stylistic fingerprints).
  • Utility Tasks: Sentiment analysis (IMDB, DynaSent), safety/moderation (Jigsaw Toxicity, DynaHate), natural language inference (ANLI, MedNLI), and detection tasks (Fraudulent email, FakeNews).

Key Insights and Results

The Correlation Gap

A critical finding of this research is that generic similarity metrics correlate poorly with actual task performance. As shown in the correlation analysis below, a high similarity score does not guarantee that the data remains useful for a specific classifier. The Kendall’s Tau correlation between generic metrics and task performance averages only around 0.4-0.5.

Kendall’s Tau correlation between generic metrics and specific task performance, highlighting the low reliability of surface-level metrics.

Task Fragility and Model Performance

Utility degradation is highly uneven across tasks. Sentiment analysis tasks proved remarkably resilient to anonymization, while Natural Language Inference (NLI) and hate speech detection were extremely fragile. For example, MedNLI performance dropped significantly across almost all tested models.

LLMs emerged as the most effective anonymizers for both PII and authorship. For PII redaction, models like Llama-3.1-8B achieved up to 99% recall. However, LLMs often struggle with anonymizing hate speech datasets due to built-in safety filters, which can lead to over-redaction and utility drops as high as 42%.

Interestingly, model size plays a counter-intuitive role in privacy: in authorship obfuscation, smaller models (e.g., Llama-3.1-8B) often provide better privacy than larger versions (70B) because they modify the original text more aggressively, albeit at a higher cost to utility.

Conclusion and Future Directions

While TAU-EVAL provides a robust baseline for English text, future work is needed to address multilingual anonymization challenges. Additionally, utility loss measurements currently depend on the specific task model used (e.g., ModernBERT-large). The authors suggest that "task-aware" anonymization—where the model is prompted with the intended use of the text—could potentially shift the privacy-utility trade-off curve toward more favorable outcomes.

Summary updated

Abstract

Text anonymization is the process of removing or obfuscating information from textual data to protect the privacy of individuals.This process inherently involves a complex trade-off between privacy protection and information preservation, where stringent anonymization methods can significantly impact the text's utility for downstream applications.Evaluating the effectiveness of text anonymization proves challenging from both privacy and utility perspectives, as there is no universal benchmark that can comprehensively assess anonymization techniques across diverse, and sometimes contradictory contexts.We present TAU-EVAL, an open-source framework for benchmarking text anonymization methods through the lens of privacy and utility task sensitivity.A Python library 1 , code, documentation and tutorials 2 are publicly available.
BibTeX
@misc{loiseau2025tau,
  author = {Gabriel Loiseau and Damien Sileo and Damien Riquet and Maxime Meyer and Marc Tommasi},
  title = {{Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization}},
  year = {2025},
  eprint = {2506.05979},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05979},
}

Loading related content...

Figure