Ping Yu, Yang Zhao, Chunyuan Li, Changyou Chen Findings of the Association for Computational Linguistics: EMNLP 2021
Abstract
Text Style Transfer (TST) evaluation currently suffers from a lack of standardization, often relying on single-checkpoint metrics that fail to capture the inherent trade-offs between style accuracy, content preservation, and naturalness. This paper proposes a robust evaluation framework using trade-off curves and a novel graph-based metric, "Attribute Hit," which leverages dependency parsing to accurately track content preservation. Experimental results demonstrate that these methods provide a more reliable assessment of model performance and that the proposed Attribute Loss regularization significantly improves state-of-the-art TST models.
The Evaluation Crisis in TST
Text Style Transfer aims to alter the sentiment of a sentence (e.g., negative to positive) while maintaining its core semantic content. Despite significant progress, measuring success remains a "wild west" scenario. Researchers typically report performance based on a single "best" checkpoint, focusing on three metrics: transfer accuracy, BLEU (for content), and Perplexity (for naturalness).
The authors identify three fundamental flaws in this approach:
- The Checkpoint Fallacy: Model performance varies significantly during training. Selecting a single point fails to represent a model's true capability or the necessary trade-offs it makes between competing objectives.
- BLEU is Blind: BLEU measures n-gram overlap. However, style transfer requires changing words. BLEU often rewards models that fail to transfer style (high overlap with source) while penalizing successful transfers that correctly replace style-laden words.
- Noisy Human Evaluation: Human assessments are often inconsistent and difficult to replicate, making them a shaky ground-truth for model comparison.
Proposed Methodology
To address these issues, the researchers introduced a multi-faceted framework designed for both better evaluation and improved training.
Robust Evaluation Curves
Instead of single data points, the authors advocate for plotting curves that visualize the trade-off between Style Transfer vs. Naturalness and Content Preservation vs. Naturalness. This allows for a "Pareto-style" comparison where one model can be shown to dominate another across all levels of fluency.
Graph-Based Content Extraction and "Attribute Hit"
To move beyond the limitations of BLEU, the authors developed a method to isolate core content from stylistic attributes. Using the UDify dependency parser and the VADER sentiment classifier, sentences are decomposed into:
- Attribute-Independent Words: The core content (e.g., "pizza", "service").
- Attribute-Dependent Words: The style markers (e.g., "delicious", "slow").
This decomposition enables the Attribute Hit metric, which calculates the percentage of original content words preserved in the output while ignoring the style words that are expected to change.
Attribute Loss Regularization
Beyond evaluation, the authors propose Attribute Loss, a regularization term used during training. This loss uses cosine similarity in the latent space to force the model to stay close to content-related representations while distancing itself from the original style-related embeddings.
Key Results and Insights
The framework was tested on the YELP (business reviews) and IMDB (movie reviews) datasets. Key findings include:
- Superior Correlation: The "Attribute Hit" metric correlates much more strongly with human judgment of content preservation than standard BLEU scores.
- Model Dominance: When applying Attribute Loss regularization, existing state-of-the-art models showed significant improvement. On trade-off curves, the regularized versions "dominated" the originals, achieving higher accuracy at equivalent naturalness levels.
- Structural Importance: The success of the graph-based approach suggests that linguistic structure (dependency trees) is a more robust signal for style transfer than simple word-frequency dictionaries.
Conclusion
By shifting the focus from single-metric snapshots to comprehensive trade-off curves and structurally-aware content metrics, this work provides a more rigorous foundation for Text Style Transfer research. While the method currently relies on the accuracy of external dependency parsers, it represents a significant step toward objective and reproducible evaluation in the field.
Summary updated