Automatic Prompt Rewriting for Personalized Text Generation

Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, Michael Bendersky

Abstract

Personalizing large language models (LLMs) is challenging when models are accessed via "frozen" APIs that do not allow for weight fine-tuning. This paper introduces a Prompt Rewriter, a T5-based model designed to automatically optimize discrete text prompts for better personalization. Using a hybrid training approach that combines supervised learning (SL) on synthetic labels with reinforcement learning (RL) via Proximal Policy Optimization (PPO), the system learns to filter noise and synthesize stylistic cues. Experimental results across email, product reviews, and social media datasets demonstrate that this automated rewriting significantly outperforms manual engineering and standard retrieval-based baselines.

Introduction

The rise of powerful LLMs like GPT-4 and Gemini has shifted the paradigm of NLP from model fine-tuning to prompt engineering. For tasks like personalized text generation—where the goal is to mimic a specific user's writing style—this shift presents a hurdle. Users cannot modify the internal parameters of these "frozen" models. Current solutions often rely on "stuffing" prompts with raw user data, which frequently results in noisy, suboptimal inputs that confuse the model. Manual prompt engineering is a partial solution but is tedious and difficult to scale across diverse users and domains.

Methodology: The SL-RL Chain

To address these limitations, the authors propose a specialized Prompt Rewriter. The rewriter acts as an intermediary, transforming raw user context into an optimized prompt that maximizes the output quality of the frozen LLM.

The three-stage training process for the Prompt Rewriter: (a) Label generation via randomization, (b) Supervised learning on the best-performing variants, and (c) Reinforcement learning using generation performance as a reward.

The training involves a two-step "SL-RL" chain:

  1. Supervised Learning (SL): To bootstrap the model, the authors generated synthetic training data. By creating multiple variants of an original prompt (e.g., by shuffling or pruning keywords) and testing them against the frozen LLM, they identified "best" prompts based on BLEU scores. These served as gold labels to train the initial rewriter, narrowing the massive search space of potential prompts.
  2. Reinforcement Learning (RL): To enable the rewriter to explore beyond simple permutations, it was fine-tuned using Proximal Policy Optimization (PPO). The reward signal was derived directly from the frozen LLM's performance on the rewritten prompt. This allows the rewriter to learn stylistic additions and complex structural changes that were not present in the initial synthetic data.

Crucially, this method optimizes discrete (hard) prompts. Unlike "soft prompting," which requires access to internal model embeddings, these rewritten prompts are plain text, making them fully compatible with any black-box API.

Key Insights and Ideas

Handling Sensitivity to Noise

A critical finding was that frozen LLMs are highly sensitive to "noise" in the prompt. While a fine-tuned model might learn to ignore irrelevant context, a frozen model often performs better when the rewriter removes messy summaries and focuses strictly on keywords and stylistic descriptors. In many cases, the rewriter learned to completely prune the summary section to improve clarity.

Style Synthesis

The researchers found that "Writing Style Synthesis"—asking an LLM to describe a user's style in natural language (e.g., "professional yet concise" or "sarcastic")—was more effective than providing raw examples of past writing. The rewriter learned to prioritize these synthesized descriptions, which proved more effective than raw data stuffing.

Human-Readable Optimization

Unlike many RL-optimized prompts that can devolve into gibberish, these rewritten prompts remained human-readable. This transparency allows users to understand the strategies the AI is using to achieve better results.

Results and Evaluation

The method was evaluated on three datasets: the Avocado Research Email Collection, Amazon Book Reviews, and Reddit comments. The hybrid SL-RL approach consistently outperformed all baselines:

  • Significant Gains: On the Reddit dataset, the BLEU score improved from a baseline of 13.13 to 20.59 using the rewritten prompts.
  • Synergy: The study highlighted the synergy between the two learning stages. RL provided the necessary exploration to find novel prompt structures, while SL provided the stability needed to keep the search space manageable.

Conclusion

Automatic prompt rewriting offers a robust path forward for personalizing black-box LLMs. By treating prompt optimization as a sequence-to-sequence task trained via RL, researchers can achieve high-quality, personalized outputs without needing access to model weights. Future work will look toward more sophisticated reward metrics beyond BLEU and expanding the scope of rewriting to include core instructions.

Summary updated

BibTeX
@inproceedings{li2024automatic,
  author = {Cheng Li and Mingyang Zhang and Qiaozhu Mei and Weize Kong and Michael Bendersky},
  title = {{Automatic Prompt Rewriting for Personalized Text Generation}},
  booktitle = {The Web Conference},
  year = {2024},
  doi = {10.1145/3589334.3645408},
}
Figure