"Transforming" Delete, Retrieve, Generate Approach for Controlled Text Style Transfer

Akhilesh Sudhakar, Bhargav Upadhyay, Arjun MaheswaranEMNLP 2019 · DOI

TL;DR

This paper introduces the Generative Style Transformer (GST), a novel approach to text style transfer that leverages pre-trained language models and the Transformer architecture. It improves upon existing "Delete, Retrieve, Generate" (DRG) frameworks by using a Transformer-based Delete Transformer (DT) and a decoder-only Transformer (GST) for generation, achieving state-of-the-art results on sentiment, gender, and political slant transfer tasks without requiring parallel corpora.

What Problem Does This Paper Address?

Text style transfer aims to rewrite sentences to match a target style (e.g., sentiment, formality) while preserving the original content. A major hurdle in this field is the scarcity of parallel corpora – datasets where sentences are identical in content but differ only in style. Existing methods often rely on adversarial training with latent representations, which can be difficult to train, slow to converge, and produce low-quality outputs. Furthermore, these methods often lack fine-grained control over the target style attributes.

Methods

The authors build upon the "Delete, Retrieve, Generate" (DRG) framework. Their key innovations lie in replacing components of the original DRG with Transformer-based models:

  • Delete Transformer (DT): Instead of a standard LSTM-based deletion mechanism, they use a BERT-based Transformer as a style classifier. By analyzing attention weights, they identify and remove words that contribute most to the source style, aiming to preserve content.
  • Generative Style Transformer (GST): This is a decoder-only Transformer, similar to GPT, used for the generation phase. It takes the content (from the deletion step) and optionally retrieved target style attributes to generate a new sentence. This approach moves away from sequence-to-sequence encoder-decoder models.

They propose two variants of GST:

  • Blind Generative Style Transformer (B-GST): Generates text based on content and target style, without explicit attribute guidance.
  • Guided Generative Style Transformer (G-GST): Generates text using content and explicitly retrieved target attributes, allowing for more fine-grained control.

Datasets

The authors evaluate their models on five datasets commonly used in style transfer research:

  • YELP: Business reviews with positive/negative sentiment labels.
  • AMAZON: Product reviews with positive/negative sentiment labels.
  • CAPTIONS: Image captions labeled as factual, romantic, or humorous.
  • POLITICAL: Facebook comments from US Senators and Representatives, labeled as Democrat or Republican.
  • GENDER: Yelp reviews labeled by gender (male/female).

They retain the train-dev-test splits used in previous works (Li et al., 2018; Prabhumoye et al., 2018).

Key Insights and Ideas

  • Transformer Attention for Deletion: Leveraging the attention mechanism of Transformers for style classification provides a more robust way to identify and remove style-specific words compared to previous methods.
  • Decoder-Only Transformer for Generation: Using a GPT-like decoder-only Transformer for generation allows for more fluent and contextually aware sentence creation, moving beyond the limitations of LSTM-based decoders.
  • Fine-grained Control with G-GST: The G-GST variant offers significant control by allowing manual specification of target attributes, which is crucial for real-world applications where precise style manipulation is needed.
  • GLEU as a Better Metric: The paper proposes Generalized Language Evaluation Understanding Metric (GLEU) as a more suitable automatic evaluation metric for style transfer than BLEU, as it considers the source sentence, generated sentence, and reference sentence, better capturing localized edits.

Key Results

  • State-of-the-Art Performance: B-GST and G-GST outperform existing state-of-the-art models on sentiment, gender, and political slant transfer tasks across multiple datasets, as judged by human evaluations.
  • Improved Deletion: Their Delete Transformer (DT) deletes style attributes more effectively and wrongly deletes non-style attributes less often than the D&R model from Li et al. (2018).
  • Sentence Length Correlation: Generated sentence lengths from B-GST closely correlate with source sentence lengths, unlike the D&R model.
  • High BLEU Scores: The models achieve high BLEU scores, suggesting good preservation of non-stylistic content.

Limitations and Future Work

  • Retrieve Component in G-GST: The authors acknowledge that the G-GST model performs worse than B-GST due to limitations in its Retrieve component. Improving this component is a key area for future work.
  • Automatic Metrics: While GLEU shows promise, the paper notes that automatic metrics (like BLEU and perplexity) do not always correlate perfectly with human evaluations, and a detailed statistical correlation study for GLEU is left for future work.
  • Real-world Applicability: While G-GST offers control, further improvements to the retrieval mechanism are needed for broader real-world applications.

Summary updated

BibTeX
@inproceedings{sudhakar2019transforming,
  author = {Akhilesh Sudhakar and Bhargav Upadhyay and Arjun Maheswaran},
  title = {{"Transforming" Delete, Retrieve, Generate Approach for Controlled Text Style Transfer}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1322},
}
Figure