Authorship Style Transfer with Policy Optimization

Shuai Liu, Shantanu Agarwal, Jonathan MayarXiv 2024 · DOI · arXiv

In the era of Large Language Models (LLMs), generated text often carries a distinct, generic "AI flavor." While these models are highly capable, making them write with the specific stylistic fingerprint of an individual—or even a specific community—remains a challenge, especially when only a few examples of the target style are available.

Researchers from the University of Southern California’s Information Sciences Institute have introduced ASTRAPOP (Authorship Style TRAnsfer with Policy OPtimization), a framework designed to achieve high-fidelity style transfer in low-resource settings. By moving beyond simple prompting and leveraging policy optimization, ASTRAPOP enables models to adopt specific styles while preserving the original meaning of the text.

The Challenge of Authorship Style Transfer

Authorship Style Transfer (AST) involves modifying the stylistic elements of a text—such as word choice, syntax, and punctuation—without altering its core content. Traditionally, this task required massive parallel datasets (e.g., thousands of sentences written in both the source and target styles). However, in real-world scenarios, target authors often have limited data available, such as a handful of blog posts or tweets.

While In-Context Learning (ICL) can help, it often succeeds only at stripping away the original style without effectively adopting the new one. ASTRAPOP addresses this gap through a robust two-stage training process.

The ASTRAPOP Framework

ASTRAPOP utilizes a "tune-and-optimize" approach that combines supervised learning with reinforcement learning principles:

  1. Stage 1: Supervised Fine-Tuning (SFT): To overcome the lack of parallel data, the researchers generate pseudo-parallel data. A "neutral paraphraser" (GPT-3.5) is used to strip the style from target texts. A reference model is then trained to perform the reverse operation: transforming neutral text back into the target style.
  2. Stage 2: Policy Optimization (PO): This stage refines the model using a Style Critic (reward model). The reward function is designed to push the model TOWARD the target style and AWAY from the source style. To ensure stability and efficiency, the framework employs RL-free algorithms like Direct Preference Optimization (DPO) and Contrastive Preference Optimization (CPO).
The ASTRAPOP framework architecture, illustrating the two-stage process of supervised fine-tuning and policy optimization.

Experimental Results

The framework was evaluated on two scales: individual authorship (using the Million User Dataset) and community-level style (using the ETS Corpus of Non-Native Written English).

ASTRAPOP consistently outperformed state-of-the-art baselines like STYLL and STEER. Notably, in the community-level (L1) transfer task, ASTRAPOP-CPO achieved a "Toward" score of 0.655, nearly doubling the performance of previous models. The use of RL-free optimization proved critical, offering greater stability and better results than traditional PPO-based methods.

Performance comparison on the community-level (L1) transfer task, highlighting ASTRAPOP-CPO's significant gains in style accuracy.

Key Insights and Future Directions

  • Balanced Rewards: Explicitly rewarding the model for moving away from the source style is essential to eliminate "style residue."
  • Efficiency: RL-free methods (DPO/CPO) are not only more stable but also computationally faster, avoiding the heavy over-generation steps required by earlier methods.
  • Ethical Considerations: While AST can be used for authorship privatization (protecting identity), it also carries risks of malicious mimicking. Future work aims to incorporate safety constraints into the reward function to prevent the generation of toxic content in a target's style.

By bridging the gap between simple prompting and complex reinforcement learning, ASTRAPOP provides a scalable path toward truly personalized AI writing assistants.

Summary updated

Abstract

Authorship style transfer aims to rewrite a given text into a specified target while preserving the original meaning in the source. Existing approaches rely on the availability of a large number of target style exemplars for model training. However, these overlook cases where a limited number of target style examples are available. The development of parameter-efficient transfer learning techniques and policy optimization (PO) approaches suggest lightweight PO is a feasible approach to low-resource style transfer. In this work, we propose a simple two-stage tune-and-optimize technique for low-resource textual style transfer. We apply our technique to authorship transfer as well as a larger-data native language style task and in both cases find it outperforms state-of-the-art baseline models.
BibTeX
@misc{liu2024authorship,
  author = {Shuai Liu and Shantanu Agarwal and Jonathan May},
  title = {{Authorship Style Transfer with Policy Optimization}},
  year = {2024},
  eprint = {2403.08043},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.08043},
}
Figure