Introduction
Text Style Transfer (TST) is the task of rewriting a sentence to change its stylistic attributes—such as sentiment or formality—while preserving its core semantic meaning. A significant challenge in TST is the lack of parallel corpora, necessitating unsupervised learning approaches. Existing methods often struggle with "disentanglement" (separating style from content) or fail to provide fine-grained control over the intensity of the transferred style. Furthermore, complex linguistic phenomena like code-switching (e.g., mixing Hindi and English) have remained largely unexplored in the TST literature.
This paper introduces VT-STOWER (VAE based Text Style Transfer with pivOt Words Enhancement leaRning), a novel framework that leverages Variational Autoencoders (VAEs) and pre-trained language models to address these gaps. By incorporating external style embeddings and a targeted "pivot words masking" strategy, VT-STOWER achieves state-of-the-art performance and enables tunable style strength.
Methodology: VT-STOWER
The VT-STOWER architecture utilizes pre-trained language models (like RoBERTa or XLM-R) to generate contextualized representations. The training process is divided into two distinct stages to ensure the model learns both the latent content distribution and the specific markers of style.
Stage I: VAE and Style Embeddings
In the first stage, the model co-trains a VAE alongside external style embeddings. Rather than attempting to strip style from the latent space, the model learns vector representations for specific styles (e.g., "positive" vs. "negative"). These embeddings are used to map the transition between the source and target styles during the generation process.
Stage II: Pivot Words Enhancement
The core innovation of VT-STOWER is the "pivot words masking" technique. The authors use a separate style classifier to identify "pivot words"—tokens that strongly indicate a sentence's style. During training, these words are probabilistically masked. The VAE is then tasked with reconstructing the original sentence using the frozen style embeddings from Stage I. This forces the decoder to rely on the style embeddings to fill in stylistic gaps while preserving the "anchor words" that carry the sentence's primary meaning.
Experimental Setup
The model was evaluated on three diverse datasets:
- Sentiment Transfer (Yelp): Restaurant reviews categorized as positive or negative.
- Formality Transfer (GYAFC): Informal vs. formal sentences from the Family and Relationships domain.
- Code-Switching Transfer (LinCE): A novel application involving Hindi ↔ Hinglish transfer. This task is particularly challenging due to its low-resource nature and the unique linguistic rhythms of mixed-language speech.
Results and Discussion
VT-STOWER demonstrated superior performance across all tasks compared to state-of-the-art baselines like Style Transformer and StyIns.
- Sentiment: On the Yelp dataset, VT-STOWER achieved 91.1% accuracy with a perplexity (PPL) of 30.78, significantly outperforming the fluency of previous high-accuracy models.
- Formality: The model reached 81.0% accuracy on GYAFC, a substantial improvement over the Style Transformer's 56.05%.
- Code-Switching: VT-STOWER successfully balanced style transfer and linguistic integrity, achieving 68.70% accuracy and 26.42 BLEU, whereas baselines often sacrificed sentence structure for style accuracy.
Tunable Style Strength
A key feature of VT-STOWER is the ability to adjust the style weight ($w$) during inference. By scaling the target style embedding, users can control the intensity of the transfer. However, results indicate a clear trade-off: as style weight increases, transfer accuracy improves, but fluency (PPL) and content preservation (BLEU) tend to decline.
Conclusion
VT-STOWER represents a significant advancement in unsupervised text style transfer. By combining VAEs with pivot-word-aware masking, the model effectively captures stylistic nuances even in complex, low-resource scenarios like code-switching. The ability to tune style strength provides a practical tool for controlled text generation. Future work will explore extending this framework to broader multilingual contexts and further refining the balance between style intensity and semantic preservation.
Summary updated