Text Style Transfer: A Review and Experimental Evaluation

Zhiqiang Hu, Roy Ka-Wei Lee, Charu C. Aggarwal, Aston Zhang

TL;DR

This paper provides a comprehensive review and experimental evaluation of Text Style Transfer (TST) techniques. It categorizes existing methods, analyzes their strengths and weaknesses, and presents a large-scale reproducibility study benchmarking 19 state-of-the-art algorithms on sentiment and formality transfer tasks. The work highlights the challenges in TST, particularly in disentangling content from style and in developing robust evaluation metrics.

What Problem Does This Paper Address?

Text Style Transfer (TST) aims to alter the stylistic properties of text (e.g., formality, sentiment, author's voice) while preserving its core meaning. While the field has seen rapid development, the literature is disparate, making it difficult to understand the landscape of existing algorithms and their comparative performance. This paper addresses this by providing a structured overview, a taxonomy of methods, and a crucial empirical evaluation to benchmark and compare various TST approaches. It also highlights the ongoing challenges in evaluating TST effectively and the need for more robust methodologies.

Methods

The paper categorizes TST methods based on three main criteria:

  • Data Settings: Parallel Supervised, Non-Parallel Supervised, and Purely Unsupervised.
  • Strategies: Explicit Style-Content Disentanglement, Implicit Style-Content Disentanglement, and Without Disentanglement.
  • Techniques: This includes Sequence-to-Sequence models, Explicit Style Keyword Replacement, Adversarial Learning, Back-Translation, Attribute Control Generation, Entangled Latent Representation Editing, Reinforcement Learning, and Purely Unsupervised Methods.

The authors also introduce a comprehensive reproducibility study where they implement and benchmark 19 TST models using standard evaluation metrics like Style Transfer Accuracy (ACC), BLEU, sBLEU, Word Overlap (WO), and perplexity (PPL).

Datasets

The paper discusses and utilizes several datasets for training and evaluation:

  • Yelp: A dataset of restaurant reviews, commonly used for sentiment transfer.
  • GYAFC (Grammarly's Yahoo Answers Formality Corpus): A parallel dataset specifically designed for formality transfer.
  • Shakespearean-Modern: A parallel dataset for author imitation style transfer.
  • IMDb: A movie reviews dataset.
  • Amazon: A product review dataset.
  • PNTD (Paper-News Title Dataset): Used for paper-news title transfer.
  • Caption Dataset: Used for transferring factual captions to romantic and humorous styles.
  • Gender Dataset: Derived from Yelp reviews, annotated with reviewer gender.
  • Political Dataset: Collected from Facebook posts of US Senators and House members.
  • Twitter and Reddit: Used for offensive language correction.

Key Insights and Ideas

  • Formality is Harder than Sentiment: The study reveals that formality transfer is significantly more challenging than sentiment transfer, with average style transfer accuracy on the GYAFC dataset being much lower than on Yelp.
  • No Single Best Model: The reproducibility study demonstrates that no single TST model excels across all evaluation metrics (transfer strength, content preservation, fluency). This suggests a trade-off inherent in TST tasks.
  • Explicit Keyword Replacement Works Well for Sentiment: Methods like DeleteOnly, Template, and Del&Retri performed well on sentiment transfer, likely because sentiment is often encoded in specific keywords that can be directly replaced.
  • Content Preservation is a Major Hurdle: Many methods, especially those involving implicit style-content disentanglement, struggle with preserving the original content, leading to low sBLEU scores.
  • Human Evaluation Remains Crucial: Despite automated metrics, human evaluation is vital for assessing the nuanced aspects of TST, as style perception can be subjective.

Key Results

The reproducibility study on Yelp and GYAFC datasets yielded several key findings:

  • Sentiment Transfer: Models like DualRL, PTO, B-GST, and PFST showed a good balance across metrics. Explicit style keyword replacement methods generally performed well.
  • Formality Transfer: Models like Ctrl-Gen, DAST, and DAST-C achieved good performance in both style transfer accuracy and content preservation. Methods without style-content disentanglement generally performed best on this task.
  • Trade-offs: The analysis consistently showed an inverse relationship between transfer strength and content preservation across models and tasks.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Style-Content Disentanglement: This remains an open research question. Future work should focus on designing experiments to quantify disentanglement and exploring it across various TST tasks.
  • Style Embeddings: A deeper analysis of learned style representations is needed to understand what information they capture beyond just style labels.
  • Unsupervised TST: Developing methods that require little to no labeled data is a promising direction, building on recent work that uses semantic relatedness and readability scores.
  • Beyond Binary Transfer: Exploring more dynamic tasks like multiple-attribute style transfer and domain-aware TST is crucial for real-world applications.
  • Cross-Lingual TST: Most existing work focuses on English. Research into TST for other languages, considering their unique stylistic properties, is needed.
  • Automatic Evaluation: The limitations of current metrics, especially for formality transfer, necessitate the development of novel automatic evaluation metrics.
  • Ethical Considerations: The paper highlights potential negative use-cases like content manipulation and forgery, emphasizing the need for ethical review processes in TST research.

Summary updated

Abstract

The stylistic properties of text have intrigued computational linguistics researchers in recent years. Specifically, researchers have investigated the Text Style Transfer (TST) task, which aims to change the stylistic properties of the text while retaining its style independent content. Over the last few years, many novel TST algorithms have been developed, while the industry has leveraged these algorithms to enable exciting TST applications. The field of TST research has burgeoned because of this symbiosis. This article aims to provide a comprehensive review of recent research efforts on text style transfer. More concretely, we create a taxonomy to organize the TST models and provide a comprehensive summary of the state of the art. We review the existing evaluation methodologies for TST tasks and conduct a large-scale reproducibility study where we experimentally benchmark 19 state-of-the-art TST algorithms on two publicly available datasets. Finally, we expand on current trends and provide new perspectives on the new and exciting developments in the TST field.
BibTeX
@article{hu2020text,
  author = {Zhiqiang Hu and Roy Ka-Wei Lee and Charu C. Aggarwal and Aston Zhang},
  title = {{Text Style Transfer: A Review and Experimental Evaluation}},
  journal = {SIGKDD Explor.},
  year = {2020},
}
Figure