Fluency Matters! Controllable Style Transfer with Syntax Guidance

Ji-Eun Han, Kyung-Ah SohnACL 2023

TL;DR

This paper tackles the challenge of controllable text style transfer, where the goal is to change a text's style (e.g., sentiment) while preserving its content and ensuring natural fluency. The authors propose a novel approach that incorporates syntactic parsing information to guide the generation process, leading to more fluent and natural-sounding transferred text, especially at varying degrees of style control.

What Problem Does This Paper Address?

Text style transfer, the task of altering a text's stylistic attributes (like sentiment, formality, or politeness) without changing its core meaning, has numerous applications, from data augmentation to marketing copy generation. A key challenge in this field is controllable style transfer, where users can specify the degree to which the style should be altered. While existing methods can achieve this control, they often suffer from a trade-off: as the style transfer becomes more pronounced (to increase accuracy), the fluency and naturalness of the generated text degrade. This means the output can sound awkward or even nonsensical. This paper aims to bridge this gap by improving the fluency of controllable style transfer, ensuring that the generated text remains natural and human-like, regardless of the desired style intensity.

Methods

The core of the proposed method lies in incorporating syntax parsing information into the style transfer process. The authors build upon an autoencoder-based architecture, which typically consists of a semantic encoder, a style selector, and a decoder. Their novel contribution is the addition of a syntactic encoder.

Here's a breakdown of their approach:

  • Semantic Encoder: Encodes the input text into a semantic embedding. Notably, they don't use positional encoding here, aiming to capture meaning more like a "bag of words" representation, which they argue can be effective for meaning preservation.
  • Syntactic Encoder: This is the key innovation. It takes the linearized syntax parse of the input text (derived from constituency parse trees) and generates a syntactic embedding. They use a Transformer encoder with positional encoding for this, ensuring the order of syntactic elements is captured.
  • Style Selector: This component determines the target style. It calculates similarities between style embeddings (positive and negative) and the semantic embedding, predicts the style label, and then selects the appropriate style embedding.
  • Decoder: The decoder takes a concatenated embedding of the semantic, syntactic, and style information and autoregressively generates the output text.

During inference, the semantic embedding is adjusted by adding a weighted style embedding. The style transfer weight (w) controls the degree of style transfer, and the authors demonstrate that their model effectively separates semantic embeddings based on this weight.

Datasets

The paper evaluates their model on two commonly used datasets for unsupervised text style transfer:

  • Yelp: Restaurant reviews, labeled as positive or negative based on scores.
  • Amazon: Product reviews, also labeled as positive or negative.

They use preprocessed versions of these datasets, noting the number of samples for training, validation, and testing.

Key Insights and Ideas

  • Syntax as a Fluency Anchor: The central insight is that syntactic structure acts as a crucial constraint for maintaining fluency. By explicitly encoding and integrating syntactic information, the model is guided to produce grammatically sound and natural sentences, even when significantly altering the style.
  • Controllable Fluency: Unlike previous methods that see fluency degrade with increased style transfer, their approach maintains stable perplexity (a measure of fluency) across different style transfer strengths. This is a significant improvement for controllable style transfer.
  • Disentangling Meaning from Style: While not a strict disentanglement approach, the use of a "bag-of-words" like semantic embedding and the explicit syntactic encoder helps to separate content from style more effectively, leading to better content preservation.
  • Style Weight as a Direct Control: The style transfer weight w directly influences the separation of semantic embeddings in a visualization, providing a clear mechanism for controlling the degree of style transfer.

Key Results

  • Fluency Improvement: The proposed model consistently achieves the lowest perplexity (PPL) scores among all compared models on both Yelp and Amazon datasets, often coming close to human reference scores. For example, on Yelp, their PPL is 40.9, compared to a human reference of 42.3.
  • Competitive Accuracy: While prioritizing fluency, the model still achieves competitive accuracy in transferring the target style, outperforming some baselines and being comparable to others.
  • Syntax Preservation: An ablation study and a dedicated syntax preservation evaluation show that their method significantly preserves the syntactic structure of the original text, outperforming baselines in this regard.
  • Qualitative Examples: The paper provides qualitative examples showing how their model can transform sentences (e.g., "gross" to "good") while preserving content, and how other models sometimes introduce extraneous words.

Limitations and Future Work

  • BLEU Score: The authors acknowledge that their model yields lower BLEU scores compared to some other controllable style transfer models. They hypothesize this is because BLEU is based on a single human reference, and their model's ability to generate diverse sentences might not always overlap with that specific reference.
  • Content Preservation: While generally good, they note that future work aims to improve content preservation capabilities further, especially when maintaining performance across varying style transfer weights.
  • Ethical Considerations: The paper includes an ethics statement highlighting potential risks like the generation of hate speech, misinformation, or biased content, and the misuse for malicious purposes. They emphasize the need for ethical development and use.

Summary updated

BibTeX
@inproceedings{han2023fluency,
  author = {Ji-Eun Han and Kyung-Ah Sohn},
  title = {{Fluency Matters! Controllable Style Transfer with Syntax Guidance}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure