A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques

Malik H. Altakrori, Thomas Scialom, Benjamin C. M. Fung, Jackie Chi Kit CheungEMNLP 2022 · DOI

Abstract

Authorship identification tools pose significant privacy risks to anonymous writers, such as whistleblowers and activists. While authorship obfuscation techniques aim to mitigate these risks, current evaluation methodologies often rely on weak baselines and abstract metrics that fail to capture semantic preservation or the risk of misattribution. This paper introduces a comprehensive evaluation framework addressing these gaps. We demonstrate that a simple back-translation baseline frequently outperforms specialized state-of-the-art techniques in both identity masking and content preservation.

1. Introduction

The increasing sophistication of authorship attribution tools necessitates robust obfuscation methods. However, the field suffers from three primary evaluation deficiencies: the use of weak baselines, reliance on abstract metrics (e.g., BLEU, ROUGE) that do not reflect semantic integrity, and a lack of consideration for "misattribution harm"—the risk of an obfuscator making text appear as though it were written by a specific innocent party.

2. Methodology: A Multifaceted Evaluation Framework

We propose a framework evaluating three critical dimensions:

  • Evasion: Measured against a state-of-the-art cross-topic authorship identification tool (Stamatatos' "Masking" technique).
  • Content Preservation via QA: Utilizing QuestEval, a question-answering approach that assesses whether core factual content remains extractable post-obfuscation.
  • Misattribution Harm: Quantified using Shannon Entropy to determine if the classifier is genuinely uncertain or confidently (and potentially harmfully) misattributing the text to another author.

The framework was tested on four techniques: Back-Translation (BT), Lexical Substitution (LSB) via BERT, Mutant-X, and Heuristic Search (A*).

3. Experimental Setup

Evaluations were conducted on two benchmark corpora: the Extended Brennan–Greenstadt (EBG) corpus and the Reuters C50 corpus, using configurations of 5 and 10 candidate authors.

4. Results and Analysis

4.1 Evasion Performance

Back-Translation (BT) emerged as a surprisingly strong performer. On the EBG dataset, BT achieved the most significant reduction in identification accuracy, outperforming complex genetic algorithms like Mutant-X.

Table 1: Evasion results showing accuracy drops across different techniques and datasets.

4.2 Content Preservation and Metric Discrepancy

A key finding is that traditional token-based metrics like BLEU and ROUGE are deceptive. While techniques like A maintain higher word overlap, they often produce less coherent text. In contrast, BT rephrases sentences naturally, leading to lower BLEU scores but the highest QuestEval* scores (0.620), indicating superior preservation of meaning.

Table 2: Content preservation metrics comparing traditional overlap scores with the QA-based QuestEval.

4.3 Misattribution and Entropy

BT consistently resulted in the highest normalized entropy (e.g., 88.8% on EBG-10). This suggests that BT creates a "generic" style that effectively hides the author without strongly mimicking another, thereby minimizing misattribution harm.

5. Conclusion

This study highlights that simple, well-tuned baselines like back-translation can be more effective than specialized obfuscation tools. Furthermore, it underscores the necessity of semantic-aware metrics like QuestEval and entropy-based measures of misattribution for a holistic understanding of authorship privacy tools. Future work should explore human-in-the-loop evaluations and more diverse datasets to further refine these findings.

Summary updated

BibTeX
@inproceedings{altakrori2022multifaceted,
  author = {Malik H. Altakrori and Thomas Scialom and Benjamin C. M. Fung and Jackie Chi Kit Cheung},
  title = {{A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.153},
}
Figure