STEER: Unified Style Transfer with Expert Reinforcement

Skyler Hallinan, Faeze Brahman, Ximing Lu, Jaehun Jung, Sean Welleck, Yejin Choi

Text style transfer (TST) involves altering the stylistic attributes of a sentence—such as its formality, sentiment, or era—while preserving its core semantic meaning. Traditionally, TST research has focused on "one-to-one" transfers (e.g., informal to formal). However, real-world applications often require "arbitrary-to-many" capabilities, where the source style is unknown and the target style can be any of several options.

The STEER (Unified Style Transfer with Expert Reinforcement) framework addresses this challenge. Despite having only 775 million parameters—making it 226 times smaller than GPT-3—STEER significantly outperforms large-scale language models in style transfer quality and robustness.

The Limitations of Scale in Style Transfer

While large language models (LLMs) like GPT-3 demonstrate impressive zero-shot capabilities, they often struggle with nuanced style transfer for two primary reasons:

  1. Parallel Data Scarcity: There is a lack of high-quality datasets that provide the same sentence across many different styles (e.g., a sentence written in the style of Shakespeare, 1990s slang, and the Bible).
  2. The Cold-Start Problem in RL: Reinforcement Learning (RL) is effective for fine-tuning, but if a model cannot initially generate high-quality stylistic examples, it lacks the necessary "good" samples to learn from reward signals.

The STEER Framework

STEER employs a two-stage approach to overcome these hurdles: expert-guided data generation followed by a transition from offline to online reinforcement learning.

1. Expert-Guided Data Generation

To mitigate the lack of parallel data, STEER uses a "Product of Experts" (DEXPERTS) method. A base paraphraser is guided during the decoding process by style-specific language models ("experts") and source-style models ("anti-experts"). This process generates a massive "pseudo-parallel" corpus, providing the model with high-quality synthetic training examples across diverse styles.

2. Two-Stage Reinforcement Learning

STEER optimizes performance by moving from an offline "warm start" to online refinement:

  • Offline RL (QUARK): The model is first trained on the synthetic data using the QUARK algorithm. This stage teaches the model to associate specific tokens with high-reward outcomes, preventing the model from starting its learning process from a state of total ignorance.
  • Online RL with Vectorized Rewards: Once initialized, the model generates its own examples and refines its parameters based on a fine-grained reward signal. Instead of a single scalar score, STEER uses a vector of rewards representing style strength, fluency, and meaning preservation. This multi-faceted feedback prevents "reward hacking," where a model might maximize style at the expense of readability.

Evaluation and Results

The researchers evaluated STEER using the Corpus of Diverse Styles (CDS), which includes 11 distinct styles ranging from historical eras (1810s, 1890s) to literary forms (Shakespeare, James Joyce) and modern social media (Twitter, Song Lyrics).

Performance Benchmarks

On the CDS benchmark, STEER achieved an overall quality score ($V$) of 34.3, dramatically outperforming GPT-3 (14.7) and the previous state-of-the-art baseline, STRAP (14.6). Human evaluations mirrored these results, with participants rating STEER higher in style transfer strength while maintaining competitive fluency.

Robustness and Generalization

STEER also demonstrated remarkable robustness when tested on "out-of-domain" styles. In tests using the GYAFC (formality) and Yelp (sentiment) datasets—styles the model did not encounter during training—STEER remained the top performer. The only exception was the Shakespearean style, where GPT-3's massive pre-training data gave it a slight edge.

Key Insights

  • Efficiency over Scale: STEER proves that task-specific expert guidance and structured RL can be more effective than raw parameter count. A 775M parameter model (based on GPT-2 Large) can be more precise than a 175B parameter model when properly "steered."
  • Solving the Cold Start: The transition from offline to online RL ensures the model doesn't "wander" during early training phases, leading to faster and more stable convergence.
  • Fine-Grained Optimization: Vectorized rewards are crucial for balancing the inherent trade-offs in TST, ensuring that meaning and fluency are not sacrificed for stylistic flair.

Conclusion

STEER represents a significant step forward in unified style transfer. By combining synthetic data generation with sophisticated RL techniques, it achieves superior performance with a fraction of the computational resources required by LLMs. Future research may focus on "continual learning" to allow the model to adapt to new styles without retraining the entire pipeline, as well as addressing potential ethical risks associated with generative text.

Summary updated

BibTeX
@inproceedings{hallinan2023steer,
  author = {Skyler Hallinan and Faeze Brahman and Ximing Lu and Jaehun Jung and Sean Welleck and Yejin Choi},
  title = {{STEER: Unified Style Transfer with Expert Reinforcement}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.506},
}
Figure