Abstract
Text Style Transfer (TST) aims to alter the underlying style of the source text to another specific style while keeping the same content. Due to the scarcity of high-quality parallel training data, unsupervised learning has become a trending direction for TST tasks. In this paper, we propose a novel VAE based Text Style Transfer with pivOt Words Enhancement leaRning (VT-STOWER) method which utilizes Variational AutoEncoder (VAE) and external style embeddings to learn semantics and style distribution jointly. Additionally, we introduce pivot words learning, which is applied to learn decisive words for a specific style and thereby further improve the overall performance of the style transfer. The proposed VT-STOWER can be scaled to different TST scenarios given very limited and non-parallel training data with a novel and flexible style strength control mechanism. Experiments demonstrate that the VT-STOWER outperforms the state-of-the-art on sentiment, formality, and code-switching TST tasks.
1. Introduction
Text Style Transfer (TST) involves modifying stylistic attributes of a sentence—such as sentiment, formality, or language variety—while preserving its core semantic content. For instance, a model might convert a negative restaurant review into a positive one, or transform an informal text message into a formal email. The primary challenge in TST is the lack of high-quality parallel datasets where the same content is expressed in multiple styles.
To address this, researchers have turned to unsupervised learning. However, existing methods often struggle to perfectly disentangle style from content and lack flexible mechanisms to control the intensity of the style transfer. This paper introduces VT-STOWER, which leverages pivot word masking to force the model to learn effective style injection.
2. Methodology: The VT-STOWER Framework
The VT-STOWER (VAE based Text Style Transfer with pivOt Words Enhance-ment leaRning) framework utilizes Variational Autoencoders (VAEs) and external style embeddings to learn semantics and style distributions jointly. The training process is divided into two distinct stages.
2.1 Stage I: Joint Learning
In the first stage, the model uses a Transformer-based VAE to learn a latent representation $z$ of the input text. Simultaneously, it learns external style embeddings by minimizing the distance between the latent vector of a sentence and its corresponding style embedding (e.g., a "formal" vector). This establishes a shared space for content and style.
2.2 Stage II: Pivot Word Enhancement
The core innovation of VT-STOWER is the use of pivot words to enhance style injection. Pivot words are those that strongly signal a specific style, identified using the attention weights of a pre-trained style classifier.
During this stage, these pivot words are masked in the input sentence. To reconstruct the original sentence, the decoder is forced to rely on the target style embeddings to "fill in" the missing stylistic cues. This process effectively teaches the model how style and content interact and how to inject specific stylistic markers into a content-rich but style-depleted representation.
2.3 Inference and Style Control
At test time, the model performs style transfer by modifying the latent vector $z$ using a simple arithmetic operation: $$z' = z + w(s_{target} - s_{source})$$ Here, $w$ is a style weight that acts as a "dial," allowing users to control the strength of the transfer. A higher $w$ results in a more pronounced style change, providing a flexible mechanism to trade off between style accuracy and content preservation.
3. Experiments and Results
VT-STOWER was evaluated on three diverse TST tasks: Sentiment Transfer (Yelp dataset), Formality Transfer (GYAFC dataset), and Code-Switching (LinCE dataset). Code-switching, transferring between a single language (Hindi) and a multilingual mix (Hinglish), represents a particularly challenging low-resource task.
3.1 Performance Comparison
The model outperformed state-of-the-art baselines across all tasks, as measured by the Geometric Mean (GM) of accuracy, fluency (Perplexity), and content preservation (BLEU).
- Sentiment: Achieved a GM of 8.61, surpassing the previous best of 8.49.
- Formality: Reached a GM of 7.21, compared to 7.09 for the Style Transformer.
- Code-Switching: This task highlighted the model's robustness. While some baselines achieved high accuracy, they produced nearly unreadable text (high Perplexity). VT-STOWER maintained a superior balance, achieving a Perplexity of 30.02 and a GM of 8.11.
4. Discussion and Conclusion
VT-STOWER demonstrates that masking style-heavy "pivot words" is an effective strategy for unsupervised style transfer. By forcing the decoder to rely on style embeddings for reconstruction, the model learns a more robust mapping between latent content and stylistic expression. The flexible style weight mechanism provides a practical way to navigate the inherent trade-offs between style intensity, content fidelity, and fluency. Future work will explore more advanced cross-lingual pre-training to further enhance performance in low-resource scenarios like code-switching.
Summary updated