SE-DAE: Style-Enhanced Denoising Auto-Encoder for Unsupervised Text Style Transfer

Jicheng Li, Yang Feng, Jiao Ou

TL;DR

The SE-DAE (Style-Enhanced Denoising Auto-Encoder) simplifies unsupervised text style transfer by moving away from over-complex GAN and Reinforcement Learning architectures. By introducing a novel data refinement mechanism and a "style denoising" trick that treats style markers as noise to be removed, the authors achieve state-of-the-art performance using a much simpler, more stable training process.

What Problem Does This Paper Address?

Text style transfer aims to change the "style" of a sentence (e.g., making a negative review positive or a formal email informal) while keeping the core meaning intact. Because we rarely have "parallel" datasets (the same sentence written in two different styles), researchers rely on unsupervised methods.

The industry standard has been the Denoising Auto-Encoder (DAE). However, vanilla DAEs face a fundamental conflict: their job is to reconstruct the input perfectly, which often means they stubbornly "preserve" the original style words instead of transforming them. To fix this, previous researchers bolted on complex components like Generative Adversarial Networks (GANs) or Reinforcement Learning (RL). While powerful, these models are notoriously difficult to converge, require laborious tuning, and often fail to generalize to new datasets.

Methods

The authors propose SE-DAE, which sticks to the simplicity of a DAE but adds two task-oriented modules:

  1. Pseudo-parallel Data Refinement: Standard models use "back-translation" to create synthetic parallel data, but the quality is often poor. SE-DAE maintains a "candidate set" of translations from all previous training epochs. It uses a scoring function—combining Word Mover Distance (WMD) for semantic similarity and a CNN classifier for style intensity—to pick the best possible training pairs from the history of the model.
  2. Style Denoising Mechanism: This is the clever bit. The authors identify "style words" (like "horrible" or "excellent") using a simple logistic regression classifier. During training, they intentionally "pollute" the input by injecting these style words as noise. The model is then forced to learn how to "denoise" the sentence by removing or replacing these style markers, which directly trains it to perform style transfer.

Datasets

The model was evaluated on two benchmark datasets:

  • YELP: A sentiment transfer task (converting positive business reviews to negative and vice versa).
  • GYAFC (Grammarly’s Yahoo Answers Formality Corpus): A formality transfer task (converting informal text to formal and vice versa). This dataset is considered more challenging due to the prevalence of abbreviations and slang.

Key Insights and Ideas

  • The Reconstruction Conflict: The "aha moment" is realizing that a standard DAE treats rare style words as important content to be saved. By re-framing style words as "noise" to be filtered out, the authors align the DAE’s objective with the goal of style transfer.
  • Data Quality > Model Complexity: The paper demonstrates that you don't need a massive, unstable GAN if you have a high-quality, refined pseudo-parallel dataset.
  • Replacement vs. Insertion: The authors found that replacing words with style noise is much more effective than simply inserting them. Replacement forces the model to reconstruct both the meaning and the new style from partial information, whereas insertion just teaches the model to ignore certain words.

Key Results

SE-DAE significantly outperformed the vanilla DAE and remained highly competitive with (or better than) much more complex SOTA models:

  • Accuracy: On the GYAFC (formality) task, SE-DAE achieved 85.5% accuracy, a massive jump from the vanilla DAE’s 29.2%.
  • Overall Performance: Using the "G2" metric (a geometric mean of accuracy and BLEU score), SE-DAE scored 68.0 on YELP, outperforming strong baselines like DualRL (67.9) and TransformerGAN (66.2).
  • Human Evaluation: In blind tests, human judges rated SE-DAE higher than most competitors in terms of style accuracy, content preservation, and fluency, particularly noting that its outputs were more "understandable" than those from GAN-based models.

Limitations and Future Work

While SE-DAE is robust, the authors noted that the GYAFC dataset remains a challenge because it contains many typos and abbreviations. The model's performance is currently tied to the quality of the initial "style word" identification (done via logistic regression); if the classifier fails to identify subtle style markers, the "style denoising" becomes less effective.

Future work could involve more sophisticated ways to identify style-carrying linguistic structures beyond just individual words, or applying this simplified DAE approach to multi-style transfer (handling more than two styles at once).

Summary updated

Abstract

Text style transfer aims to change the style of sentences while preserving the semantic meanings. Due to the lack of parallel data, the Denoising Auto-Encoder (DAE) is widely used in this task to model distributions of different sentence styles. However, because of the conflict between the target of the conventional denoising procedure and the target of style transfer task, the vanilla DAE can not produce satisfying enough results. To improve the transferability of the model, most of the existing works combine DAE with various complicated unsupervised networks, which makes the whole system become over-complex. In this work, we design a novel DAE model named Style-Enhanced DAE (SE-DAE), which is specifically designed for the text style transfer task. Compared with previous complicated style-transfer models, our model do not consist of any complicated unsupervised networks, but only relies on the high-quality pseudo-parallel data generated by a novel data refinement mechanism. Moreover, to alleviate the conflict between the targets of the conventional denoising procedure and the style transfer task, we propose another novel style denoising mechanism, which is more compatible with the target of the style transfer task. We validate the effectiveness of our model on two style benchmark datasets. Both automatic evaluation and human evaluation show that our proposed model is highly competitive compared with previous strong the state of the art (SOTA) approaches and greatly outperforms the vanilla DAE.
BibTeX
@inproceedings{li2021se,
  author = {Jicheng Li and Yang Feng and Jiao Ou},
  title = {{SE-DAE: Style-Enhanced Denoising Auto-Encoder for Unsupervised Text Style Transfer}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=pTRP0wdTt8},
}

Loading related content...

Figure