Abstract
Traditional text style transfer models often rely on disentangling content from style in a latent space, a process that frequently leads to information loss and poor content preservation. We present the Style Transformer, a model that eschews the disentanglement paradigm in favor of a Transformer-based architecture. By utilizing self-attention and a cycle-consistent training objective, our approach achieves superior style transfer accuracy and content retention on standard benchmarks without the need for parallel data.
Introduction
Text style transfer aims to alter the stylistic attributes of a sentence while preserving its semantic core. In the absence of parallel corpora, researchers have historically turned to "disentangled latent space" models. These models attempt to map sentences into a style-independent content vector, which is then combined with a target style variable for decoding.
However, this paradigm faces significant challenges. First, true disentanglement is difficult to achieve; style information almost always leaks into the content representation. Second, the information bottleneck created by compressing a sentence into a fixed-size vector often destroys fine-grained semantic details. Finally, the authors argue that explicit disentanglement is unnecessary, as a sufficiently powerful decoder can learn to overwrite stylistic features if given an entangled representation.
Methodology
The Style Transformer architecture leverages the self-attention mechanism to maintain content preservation. Unlike RNN-based models that compress sequences into a single vector, the Transformer allows the decoder to attend directly to the source tokens, facilitating the "overwriting" of stylistic elements with target attributes while fetching semantic context.
Training Objectives
The model is trained using a GAN-like framework involving a Style Transformer Network ($f_\theta$) and a Discriminator Network ($d_\phi$). The training process incorporates three primary loss functions:
- Self-Reconstruction Loss ($\mathcal{L}_{self}$): Ensures the model can reconstruct the input $x$ when the target style matches the original style $s$.
- Cycle-Consistency Loss ($\mathcal{L}_{cycle}$): A critical component where the model transfers $x$ to a target style $\hat{s}$, then attempts to reconstruct the original $x$ from the generated output. This "round-trip" ensures that semantic content is preserved throughout the transfer.
- Style Controlling Loss ($\mathcal{L}_{style}$): Penalizes the generator if the discriminator fails to classify the generated sentence as the target style.
To handle the non-differentiable nature of discrete text generation, the authors employ "continuous decoding." Instead of sampling tokens, the softmax probability distributions are passed directly to the discriminator as weighted average embeddings, providing a stable gradient for training compared to traditional reinforcement learning methods.
Experiments and Results
The model was evaluated on the Yelp and IMDb review datasets. The IMDb dataset was specifically curated using a BERT-based filter to ensure high sentiment polarity, keeping only sentences with confidence scores above 0.9.
The Style Transformer demonstrated state-of-the-art performance, particularly in content preservation as measured by BLEU scores. On the Yelp dataset, the Conditional variant achieved a style accuracy of 93.7% with a ref-BLEU of 17.1. The Multi-class variant further improved content preservation to 20.3 ref-BLEU. Human evaluations corroborated these findings, with participants preferring the Style Transformer for both fluency and content retention over competitive baselines like DeleteAndRetrieve.
Conclusion
The Style Transformer demonstrates that explicit latent space disentanglement is not a prerequisite for effective text style transfer. By replacing the information bottleneck of traditional encoders with the flexible attention mechanism of the Transformer and enforcing cycle-consistency, the model achieves a better balance between style control and semantic preservation. Future work will explore multi-attribute transfer and the integration of back-translation to further enhance unsupervised learning.
Summary updated