VAE based Text Style Transfer with Pivot Words Enhancement Learning

Haoran Xu, Sixing Lu, Zhongkai Sun, Chengyuan Ma, Chenlei Guo

Introduction

Text Style Transfer (TST) is the task of rewriting a sentence to change its stylistic attributes—such as sentiment or formality—while preserving its core semantic meaning. A significant challenge in TST is the lack of parallel corpora, necessitating unsupervised learning approaches. Existing methods often struggle with "disentanglement" (separating style from content) or fail to provide fine-grained control over the intensity of the transferred style. Furthermore, complex linguistic phenomena like code-switching (e.g., mixing Hindi and English) have remained largely unexplored in the TST literature.

This paper introduces VT-STOWER (VAE based Text Style Transfer with pivOt Words Enhancement leaRning), a novel framework that leverages Variational Autoencoders (VAEs) and pre-trained language models to address these gaps. By incorporating external style embeddings and a targeted "pivot words masking" strategy, VT-STOWER achieves state-of-the-art performance and enables tunable style strength.

Methodology: VT-STOWER

The VT-STOWER architecture utilizes pre-trained language models (like RoBERTa or XLM-R) to generate contextualized representations. The training process is divided into two distinct stages to ensure the model learns both the latent content distribution and the specific markers of style.

Stage I: VAE and Style Embeddings

In the first stage, the model co-trains a VAE alongside external style embeddings. Rather than attempting to strip style from the latent space, the model learns vector representations for specific styles (e.g., "positive" vs. "negative"). These embeddings are used to map the transition between the source and target styles during the generation process.

Stage II: Pivot Words Enhancement

The core innovation of VT-STOWER is the "pivot words masking" technique. The authors use a separate style classifier to identify "pivot words"—tokens that strongly indicate a sentence's style. During training, these words are probabilistically masked. The VAE is then tasked with reconstructing the original sentence using the frozen style embeddings from Stage I. This forces the decoder to rely on the style embeddings to fill in stylistic gaps while preserving the "anchor words" that carry the sentence's primary meaning.

The two-stage training architecture of VT-STOWER, showing VAE training and pivot word enhancement.

Experimental Setup

The model was evaluated on three diverse datasets:

  1. Sentiment Transfer (Yelp): Restaurant reviews categorized as positive or negative.
  2. Formality Transfer (GYAFC): Informal vs. formal sentences from the Family and Relationships domain.
  3. Code-Switching Transfer (LinCE): A novel application involving Hindi ↔ Hinglish transfer. This task is particularly challenging due to its low-resource nature and the unique linguistic rhythms of mixed-language speech.

Results and Discussion

VT-STOWER demonstrated superior performance across all tasks compared to state-of-the-art baselines like Style Transformer and StyIns.

  • Sentiment: On the Yelp dataset, VT-STOWER achieved 91.1% accuracy with a perplexity (PPL) of 30.78, significantly outperforming the fluency of previous high-accuracy models.
  • Formality: The model reached 81.0% accuracy on GYAFC, a substantial improvement over the Style Transformer's 56.05%.
  • Code-Switching: VT-STOWER successfully balanced style transfer and linguistic integrity, achieving 68.70% accuracy and 26.42 BLEU, whereas baselines often sacrificed sentence structure for style accuracy.

Tunable Style Strength

A key feature of VT-STOWER is the ability to adjust the style weight ($w$) during inference. By scaling the target style embedding, users can control the intensity of the transfer. However, results indicate a clear trade-off: as style weight increases, transfer accuracy improves, but fluency (PPL) and content preservation (BLEU) tend to decline.

The trade-off between style accuracy, perplexity, and BLEU score as style weight increases for the code-switching task.

Conclusion

VT-STOWER represents a significant advancement in unsupervised text style transfer. By combining VAEs with pivot-word-aware masking, the model effectively captures stylistic nuances even in complex, low-resource scenarios like code-switching. The ability to tune style strength provides a practical tool for controlled text generation. Future work will explore extending this framework to broader multilingual contexts and further refining the balance between style intensity and semantic preservation.

Summary updated

BibTeX
@inproceedings{xu2021vae,
  author = {Haoran Xu and Sixing Lu and Zhongkai Sun and Chengyuan Ma and Chenlei Guo},
  title = {{VAE based Text Style Transfer with Pivot Words Enhancement Learning}},
  booktitle = {ICON},
  year = {2021},
  url = {https://openreview.net/forum?id=pWFUlU5xbX},
}
Figure