Rethinking Sentiment Style Transfer

Ping Yu, Yang Zhao, Chunyuan Li, Changyou ChenEMNLP 2021 · OpenReview

Ping Yu, Yang Zhao, Chunyuan Li, Changyou Chen Findings of the Association for Computational Linguistics: EMNLP 2021

Abstract

Text Style Transfer (TST) evaluation currently suffers from a lack of standardization, often relying on single-checkpoint metrics that fail to capture the inherent trade-offs between style accuracy, content preservation, and naturalness. This paper proposes a robust evaluation framework using trade-off curves and a novel graph-based metric, "Attribute Hit," which leverages dependency parsing to accurately track content preservation. Experimental results demonstrate that these methods provide a more reliable assessment of model performance and that the proposed Attribute Loss regularization significantly improves state-of-the-art TST models.

The Evaluation Crisis in TST

Text Style Transfer aims to alter the sentiment of a sentence (e.g., negative to positive) while maintaining its core semantic content. Despite significant progress, measuring success remains a "wild west" scenario. Researchers typically report performance based on a single "best" checkpoint, focusing on three metrics: transfer accuracy, BLEU (for content), and Perplexity (for naturalness).

The authors identify three fundamental flaws in this approach:

  1. The Checkpoint Fallacy: Model performance varies significantly during training. Selecting a single point fails to represent a model's true capability or the necessary trade-offs it makes between competing objectives.
  2. BLEU is Blind: BLEU measures n-gram overlap. However, style transfer requires changing words. BLEU often rewards models that fail to transfer style (high overlap with source) while penalizing successful transfers that correctly replace style-laden words.
  3. Noisy Human Evaluation: Human assessments are often inconsistent and difficult to replicate, making them a shaky ground-truth for model comparison.
Figure 2: Trade-off curves reveal model dominance that single-point metrics miss. Here, Model B is shown to be strictly superior to Model A across the entire operating range of naturalness versus accuracy.

Proposed Methodology

To address these issues, the researchers introduced a multi-faceted framework designed for both better evaluation and improved training.

Robust Evaluation Curves

Instead of single data points, the authors advocate for plotting curves that visualize the trade-off between Style Transfer vs. Naturalness and Content Preservation vs. Naturalness. This allows for a "Pareto-style" comparison where one model can be shown to dominate another across all levels of fluency.

Graph-Based Content Extraction and "Attribute Hit"

To move beyond the limitations of BLEU, the authors developed a method to isolate core content from stylistic attributes. Using the UDify dependency parser and the VADER sentiment classifier, sentences are decomposed into:

  • Attribute-Independent Words: The core content (e.g., "pizza", "service").
  • Attribute-Dependent Words: The style markers (e.g., "delicious", "slow").

This decomposition enables the Attribute Hit metric, which calculates the percentage of original content words preserved in the output while ignoring the style words that are expected to change.

Figure 5: (Left/Center) Trade-off curves for Style Transformer (ST), Deep Latent Sequence (DLS), and Fine-Grained Style Transfer (FGST). (Right) A dependency tree visualization used to separate content from style attributes.

Attribute Loss Regularization

Beyond evaluation, the authors propose Attribute Loss, a regularization term used during training. This loss uses cosine similarity in the latent space to force the model to stay close to content-related representations while distancing itself from the original style-related embeddings.

Key Results and Insights

The framework was tested on the YELP (business reviews) and IMDB (movie reviews) datasets. Key findings include:

  • Superior Correlation: The "Attribute Hit" metric correlates much more strongly with human judgment of content preservation than standard BLEU scores.
  • Model Dominance: When applying Attribute Loss regularization, existing state-of-the-art models showed significant improvement. On trade-off curves, the regularized versions "dominated" the originals, achieving higher accuracy at equivalent naturalness levels.
  • Structural Importance: The success of the graph-based approach suggests that linguistic structure (dependency trees) is a more robust signal for style transfer than simple word-frequency dictionaries.

Conclusion

By shifting the focus from single-metric snapshots to comprehensive trade-off curves and structurally-aware content metrics, this work provides a more rigorous foundation for Text Style Transfer research. While the method currently relies on the accuracy of external dependency parsers, it represents a significant step toward objective and reproducible evaluation in the field.

Summary updated

BibTeX
@inproceedings{yu2021rethinking,
  author = {Ping Yu and Yang Zhao and Chunyuan Li and Changyou Chen},
  title = {{Rethinking Sentiment Style Transfer}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=p0WXFO6ElDQ},
}
Figure