Abstract
Authorship identification tools pose significant privacy risks to anonymous writers, such as whistleblowers and activists. While authorship obfuscation techniques aim to mitigate these risks, current evaluation methodologies often rely on weak baselines and abstract metrics that fail to capture semantic preservation or the risk of misattribution. This paper introduces a comprehensive evaluation framework addressing these gaps. We demonstrate that a simple back-translation baseline frequently outperforms specialized state-of-the-art techniques in both identity masking and content preservation.
1. Introduction
The increasing sophistication of authorship attribution tools necessitates robust obfuscation methods. However, the field suffers from three primary evaluation deficiencies: the use of weak baselines, reliance on abstract metrics (e.g., BLEU, ROUGE) that do not reflect semantic integrity, and a lack of consideration for "misattribution harm"—the risk of an obfuscator making text appear as though it were written by a specific innocent party.
2. Methodology: A Multifaceted Evaluation Framework
We propose a framework evaluating three critical dimensions:
- Evasion: Measured against a state-of-the-art cross-topic authorship identification tool (Stamatatos' "Masking" technique).
- Content Preservation via QA: Utilizing QuestEval, a question-answering approach that assesses whether core factual content remains extractable post-obfuscation.
- Misattribution Harm: Quantified using Shannon Entropy to determine if the classifier is genuinely uncertain or confidently (and potentially harmfully) misattributing the text to another author.
The framework was tested on four techniques: Back-Translation (BT), Lexical Substitution (LSB) via BERT, Mutant-X, and Heuristic Search (A*).
3. Experimental Setup
Evaluations were conducted on two benchmark corpora: the Extended Brennan–Greenstadt (EBG) corpus and the Reuters C50 corpus, using configurations of 5 and 10 candidate authors.
4. Results and Analysis
4.1 Evasion Performance
Back-Translation (BT) emerged as a surprisingly strong performer. On the EBG dataset, BT achieved the most significant reduction in identification accuracy, outperforming complex genetic algorithms like Mutant-X.
4.2 Content Preservation and Metric Discrepancy
A key finding is that traditional token-based metrics like BLEU and ROUGE are deceptive. While techniques like A maintain higher word overlap, they often produce less coherent text. In contrast, BT rephrases sentences naturally, leading to lower BLEU scores but the highest QuestEval* scores (0.620), indicating superior preservation of meaning.
4.3 Misattribution and Entropy
BT consistently resulted in the highest normalized entropy (e.g., 88.8% on EBG-10). This suggests that BT creates a "generic" style that effectively hides the author without strongly mimicking another, thereby minimizing misattribution harm.
5. Conclusion
This study highlights that simple, well-tuned baselines like back-translation can be more effective than specialized obfuscation tools. Furthermore, it underscores the necessity of semantic-aware metrics like QuestEval and entropy-based measures of misattribution for a holistic understanding of authorship privacy tools. Future work should explore human-in-the-loop evaluations and more diverse datasets to further refine these findings.
Summary updated