Fair Transfer of Multiple Style Attributes in Text

Karan Dabas, Nishtha Madan, Vijay Arya, Sameep Mehta, Gautam Singh, Tanmoy Chakraborty

TL;DR

Researchers have developed STMS (Style Transfer for Multiple Styles), a neural network architecture designed to simultaneously modify multiple stylistic attributes of text—such as gender and formality—while preserving the original meaning. Unlike traditional methods that apply styles one after another and often "erase" previous changes, STMS uses a joint feedback loop to ensure all target styles are represented fairly and effectively.

What Problem Does This Paper Address?

Text style transfer (TST) aims to change the "flavor" of a sentence (e.g., making it more polite or changing the perceived gender of the author) without altering its core message. While significant progress has been made in transferring single attributes, real-world applications often require multi-dimensional changes. For instance, a chatbot might need to be both "formal" and "polite" when addressing an elderly user.

The authors identify a major pitfall in current approaches: sequential transfer. If you try to change a sentence from "female" to "male" and then from "informal" to "formal" in two separate steps, the second step often reverses the progress of the first or produces garbled text. Furthermore, datasets are often biased; for example, a dataset used to train "male" style might be overwhelmingly "informal," causing a model to inadvertently inject informality whenever it tries to sound more masculine.

Methods

The authors propose the STMS Model, which moves away from sequential pipelines in favor of a joint architecture. Key components include:

  1. Style Normalization via Back-Translation: To decouple content from style, the model uses a machine translation trick. It translates an English sentence into French and then back to English. This process tends to strip away author-specific stylistic quirks, leaving a "style-neutral" latent representation ($z$).
  2. Bias Rectifier: The team observed that the Gender dataset was heavily skewed toward informal language. They used a pre-trained formality classifier to identify and duplicate minority-class samples, balancing the dataset to prevent the model from learning unintended correlations.
  3. Multi-Classifier Feedback Loop: Instead of a single decoder, STMS uses a single decoder guided by multiple style classifiers (e.g., one for gender, one for formality). During training, these classifiers provide joint feedback to the decoder. The loss function is a weighted sum of the reconstruction loss and the individual losses from each style classifier.
Figure 1: STMS Model: A Feedback based Architecture for multi-Style Transfer problem

Datasets

The model was trained and evaluated on two primary datasets:

  • Gender Dataset: Derived from Yelp reviews, labeled as male or female based on usernames (approx. 2.89M sentences per class).
  • Formality Dataset: Part of the GYAFC (Yahoo Answers) corpus, labeled as formal or informal (approx. 108K sentences per class).

Key Insights and Ideas

  • Sequential Transfer is Ineffective: The authors prove that applying style transfers in a series is a "zero-sum game" where later steps often dominate or nullify earlier ones.
  • The Content-Style Trade-off: There is an inherent inverse relationship between style strength and content preservation. The more aggressively you change the style, the more likely you are to lose the original meaning.
  • Fairness in Style: "Fairness" here refers to the balanced representation of multiple target attributes. STMS is designed so that one style (like "male") doesn't completely overwrite another (like "formal").
  • Hidden Biases in NLP Data: The discovery that 88.1% of the Gender dataset was informal highlights why many TST models struggle; they aren't just learning gender, they are accidentally learning informality.

Key Results

STMS demonstrated superior performance in balancing multiple styles compared to sequential baselines (like Sequential seq2seq or Sequential CAE).

  • Quantitative Performance: In a "Female to Male & Formal" conversion task, STMS achieved a male style strength of 0.751 and a formal style strength of 0.241. While some sequential models achieved higher formality, they did so at the cost of significantly lower gender transfer or poor content preservation.
  • Content Preservation: STMS maintained a solid content preservation score of 0.866, outperforming the Sequential CAE baseline (0.626).
  • User Study: In a study with 15 human reviewers, 73% were satisfied with the STMS translations. Furthermore, 11 out of 15 participants felt STMS reflected both target styles (maleness and formality) better than the competition.
Figure 2: Sample of Style translation of input sen- tence(Female) into Male, Formal style on Gender Dataset

Limitations and Future Work

The authors acknowledge several remaining hurdles:

  • Mutual Bias: If datasets for different styles are fundamentally non-parallel and contain mutual biases, the model will always struggle to reach "perfect" style strength for all attributes simultaneously.
  • Loss Weighting: The weights assigned to different style classifiers in the loss function are critical and currently require manual tuning.
  • Style Erasure: Even with joint training, some models still tend to erase one style to make room for another. The authors suggest that future work should focus on models that more surgically erase only the relevant stylistic markers while leaving others intact.

Summary updated

Abstract

To preserve anonymity and obfuscate their identity on online platforms users may morph their text and portray themselves as a different gender or demographic. Similarly, a chatbot may need to customize its communication style to improve engagement with its audience. This manner of changing the style of written text has gained significant attention in recent years. Yet these past research works largely cater to the transfer of single style attributes. The disadvantage of focusing on a single style alone is that this often results in target text where other existing style attributes behave unpredictably or are unfairly dominated by the new style. To counteract this behavior, it would be nice to have a style transfer mechanism that can transfer or control multiple styles simultaneously and fairly. Through such an approach, one could obtain obfuscated or written text incorporated with a desired degree of multiple soft styles such as female-quality, politeness, or formalness. In this work, we demonstrate that the transfer of multiple styles cannot be achieved by sequentially performing multiple single-style transfers. This is because each single style-transfer step often reverses or dominates over the style incorporated by a previous transfer step. We then propose a neural network architecture for fairly transferring multiple style attributes in a given text. We test our architecture on the Yelp data set to demonstrate our superior performance as compared to existing one-style transfer steps performed in a sequence.
BibTeX
@misc{dabas2019fair,
  author = {Karan Dabas and Nishtha Madan and Vijay Arya and Sameep Mehta and Gautam Singh and Tanmoy Chakraborty},
  title = {{Fair Transfer of Multiple Style Attributes in Text}},
  year = {2019},
  eprint = {2001.06693},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.06693},
}
Figure