Abstract
Evaluating the quality of text style transfer (TST) models—specifically those designed for detoxification—is a complex task that requires balancing style accuracy, content preservation, and linguistic fluency. This study investigates the reliability of automatic metrics by comparing them against large-scale human evaluations of 14 detoxification models for the Russian language. We introduce ParaDetox, a large parallel corpus for Russian detoxification, and demonstrate that while automatic metrics can identify poorly performing models, they often fail to distinguish between high-quality outputs and are susceptible to adversarial optimization. Our findings suggest that character-level metrics like ChrF serve as surprisingly robust proxies for content preservation in morphologically rich languages.
Introduction
In Natural Language Processing (NLP), standard metrics like BLEU or BERTScore are commonly used to evaluate model performance by measuring overlap with reference texts. However, in Text Style Transfer (TST), models must fundamentally alter the "flavor" of a sentence (e.g., from toxic to neutral) while preserving its core meaning. This often necessitates significant lexical changes, rendering traditional word-overlap metrics less effective. This paper addresses the critical question: To what extent can we trust automatic proxies to judge the success of text detoxification?
Methodology: Bridging the Evaluation Gap
The researchers evaluated 14 models—comprising 10 competition entries and 4 baselines—using two parallel tracks:
- Automatic Metrics: Models were assessed using a BERT-based style accuracy (STA) classifier, LaBSE embeddings for semantic similarity (SIM), and a linguistic acceptability classifier for fluency (FL). A "Joint" score ($J_a$) was derived by multiplying these three components.
- Manual Evaluation: Using the Yandex.Toloka crowdsourcing platform, the team collected 10 annotations per sentence to ensure high reliability. Workers rated the outputs on the same three dimensions: Style, Content, and Fluency.
The ParaDetox Dataset
A significant contribution of this work is the introduction of ParaDetox, a large-scale parallel corpus for Russian detoxification. The dataset contains 8,622 sentences sourced from Russian social networks (Odnoklassniki, Pikabu, and Twitter). Crowd workers performed the initial rewrites, which were then cross-verified by independent annotators to ensure both toxicity reduction and meaning preservation.
Results: The Reliability of Automatic Metrics
The study revealed a "Better Model Paradox": while automatic metrics are effective at filtering out low-quality models, their correlation with human judgment decreases as model performance improves. This indicates that current automated tools struggle with the nuances of high-quality human-like text.
Key Findings:
- Adversarial Vulnerability: One participant model specifically optimized for automatic metrics achieved a near-perfect automatic joint score ($J_a = 0.87$) but was ranked last by humans ($J_m = 0.02$). The model produced unintelligible "gibberish" that successfully triggered the automated classifiers, highlighting a dangerous flaw in relying solely on such metrics.
- ChrF as a Proxy: For Russian, a morphologically rich language, the character n-gram F-score (ChrF) showed a stronger correlation with human joint scores than many word-level metrics, suggesting it is a more reliable proxy for content preservation.
- The Style-Content Trade-off: There is a notable negative correlation between manual style accuracy and automatic content similarity. As models more aggressively remove toxic elements, automated tools often incorrectly perceive a loss of original meaning.
Conclusion
While automatic metrics are indispensable for rapid model iteration, they cannot yet replace human evaluation in the final assessment of text style transfer. The best-performing model in this study (a cleaned-up T5 variant) achieved a manual joint score of 0.63, closely approaching human-level performance (0.65). Future work must focus on developing more robust, semantic-based metrics that are resistant to adversarial gaming and better capture the subtle requirements of human communication.
Summary updated