In the era of Large Language Models (LLMs), generated text often carries a distinct, generic "AI flavor." While these models are highly capable, making them write with the specific stylistic fingerprint of an individual—or even a specific community—remains a challenge, especially when only a few examples of the target style are available.
Researchers from the University of Southern California’s Information Sciences Institute have introduced ASTRAPOP (Authorship Style TRAnsfer with Policy OPtimization), a framework designed to achieve high-fidelity style transfer in low-resource settings. By moving beyond simple prompting and leveraging policy optimization, ASTRAPOP enables models to adopt specific styles while preserving the original meaning of the text.
The Challenge of Authorship Style Transfer
Authorship Style Transfer (AST) involves modifying the stylistic elements of a text—such as word choice, syntax, and punctuation—without altering its core content. Traditionally, this task required massive parallel datasets (e.g., thousands of sentences written in both the source and target styles). However, in real-world scenarios, target authors often have limited data available, such as a handful of blog posts or tweets.
While In-Context Learning (ICL) can help, it often succeeds only at stripping away the original style without effectively adopting the new one. ASTRAPOP addresses this gap through a robust two-stage training process.
The ASTRAPOP Framework
ASTRAPOP utilizes a "tune-and-optimize" approach that combines supervised learning with reinforcement learning principles:
- Stage 1: Supervised Fine-Tuning (SFT): To overcome the lack of parallel data, the researchers generate pseudo-parallel data. A "neutral paraphraser" (GPT-3.5) is used to strip the style from target texts. A reference model is then trained to perform the reverse operation: transforming neutral text back into the target style.
- Stage 2: Policy Optimization (PO): This stage refines the model using a Style Critic (reward model). The reward function is designed to push the model TOWARD the target style and AWAY from the source style. To ensure stability and efficiency, the framework employs RL-free algorithms like Direct Preference Optimization (DPO) and Contrastive Preference Optimization (CPO).
Experimental Results
The framework was evaluated on two scales: individual authorship (using the Million User Dataset) and community-level style (using the ETS Corpus of Non-Native Written English).
ASTRAPOP consistently outperformed state-of-the-art baselines like STYLL and STEER. Notably, in the community-level (L1) transfer task, ASTRAPOP-CPO achieved a "Toward" score of 0.655, nearly doubling the performance of previous models. The use of RL-free optimization proved critical, offering greater stability and better results than traditional PPO-based methods.
Key Insights and Future Directions
- Balanced Rewards: Explicitly rewarding the model for moving away from the source style is essential to eliminate "style residue."
- Efficiency: RL-free methods (DPO/CPO) are not only more stable but also computationally faster, avoiding the heavy over-generation steps required by earlier methods.
- Ethical Considerations: While AST can be used for authorship privatization (protecting identity), it also carries risks of malicious mimicking. Future work aims to incorporate safety constraints into the reward function to prevent the generation of toxic content in a target's style.
By bridging the gap between simple prompting and complex reinforcement learning, ASTRAPOP provides a scalable path toward truly personalized AI writing assistants.
Summary updated