TL;DR
Researchers have developed STMS (Style Transfer for Multiple Styles), a neural network architecture designed to simultaneously modify multiple stylistic attributes of text—such as gender and formality—while preserving the original meaning. Unlike traditional methods that apply styles one after another and often "erase" previous changes, STMS uses a joint feedback loop to ensure all target styles are represented fairly and effectively.
What Problem Does This Paper Address?
Text style transfer (TST) aims to change the "flavor" of a sentence (e.g., making it more polite or changing the perceived gender of the author) without altering its core message. While significant progress has been made in transferring single attributes, real-world applications often require multi-dimensional changes. For instance, a chatbot might need to be both "formal" and "polite" when addressing an elderly user.
The authors identify a major pitfall in current approaches: sequential transfer. If you try to change a sentence from "female" to "male" and then from "informal" to "formal" in two separate steps, the second step often reverses the progress of the first or produces garbled text. Furthermore, datasets are often biased; for example, a dataset used to train "male" style might be overwhelmingly "informal," causing a model to inadvertently inject informality whenever it tries to sound more masculine.
Methods
The authors propose the STMS Model, which moves away from sequential pipelines in favor of a joint architecture. Key components include:
- Style Normalization via Back-Translation: To decouple content from style, the model uses a machine translation trick. It translates an English sentence into French and then back to English. This process tends to strip away author-specific stylistic quirks, leaving a "style-neutral" latent representation ($z$).
- Bias Rectifier: The team observed that the Gender dataset was heavily skewed toward informal language. They used a pre-trained formality classifier to identify and duplicate minority-class samples, balancing the dataset to prevent the model from learning unintended correlations.
- Multi-Classifier Feedback Loop: Instead of a single decoder, STMS uses a single decoder guided by multiple style classifiers (e.g., one for gender, one for formality). During training, these classifiers provide joint feedback to the decoder. The loss function is a weighted sum of the reconstruction loss and the individual losses from each style classifier.
Datasets
The model was trained and evaluated on two primary datasets:
- Gender Dataset: Derived from Yelp reviews, labeled as male or female based on usernames (approx. 2.89M sentences per class).
- Formality Dataset: Part of the GYAFC (Yahoo Answers) corpus, labeled as formal or informal (approx. 108K sentences per class).
Key Insights and Ideas
- Sequential Transfer is Ineffective: The authors prove that applying style transfers in a series is a "zero-sum game" where later steps often dominate or nullify earlier ones.
- The Content-Style Trade-off: There is an inherent inverse relationship between style strength and content preservation. The more aggressively you change the style, the more likely you are to lose the original meaning.
- Fairness in Style: "Fairness" here refers to the balanced representation of multiple target attributes. STMS is designed so that one style (like "male") doesn't completely overwrite another (like "formal").
- Hidden Biases in NLP Data: The discovery that 88.1% of the Gender dataset was informal highlights why many TST models struggle; they aren't just learning gender, they are accidentally learning informality.
Key Results
STMS demonstrated superior performance in balancing multiple styles compared to sequential baselines (like Sequential seq2seq or Sequential CAE).
- Quantitative Performance: In a "Female to Male & Formal" conversion task, STMS achieved a male style strength of 0.751 and a formal style strength of 0.241. While some sequential models achieved higher formality, they did so at the cost of significantly lower gender transfer or poor content preservation.
- Content Preservation: STMS maintained a solid content preservation score of 0.866, outperforming the Sequential CAE baseline (0.626).
- User Study: In a study with 15 human reviewers, 73% were satisfied with the STMS translations. Furthermore, 11 out of 15 participants felt STMS reflected both target styles (maleness and formality) better than the competition.
Limitations and Future Work
The authors acknowledge several remaining hurdles:
- Mutual Bias: If datasets for different styles are fundamentally non-parallel and contain mutual biases, the model will always struggle to reach "perfect" style strength for all attributes simultaneously.
- Loss Weighting: The weights assigned to different style classifiers in the loss function are critical and currently require manual tuning.
- Style Erasure: Even with joint training, some models still tend to erase one style to make room for another. The authors suggest that future work should focus on models that more surgically erase only the relevant stylistic markers while leaving others intact.
Summary updated