Learning to Rewrite Prompts for Personalized Text Generation

Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, Michael Bendersky

TL;DR

When using frozen, API-only large language models (LLMs) for personalized text generation, static or noisy personal contexts degrade output quality. This paper introduces an automatic prompt-rewriting framework that chains supervised learning (SL) with reinforcement learning (RL) to dynamically optimize input-dependent context—specifically user summaries, style profiles, and keywords. By teaching a smaller model (T5) to rewrite prompts targeting a frozen generator (PaLM 2), the authors achieve dramatic generation quality improvements across emails, product reviews, and social media comments.

What Problem Does This Paper Address?

Personalized text generation aims to match an author's specific voice, topical interests, and stylistic habits. Recent frameworks (such as FtPersLlm) build rich prompt contexts containing retrieved past documents, extracted summaries, and keywords, but rely on fine-tuning the target LLM.

However, in most real-world applications, production LLMs are accessible only via black-box APIs (e.g., ChatGPT, PaLM 2) with frozen weights and non-differentiable generation metrics (like BLEU). While fine-tuned LLMs learn to ignore noisy context snippets, frozen LLMs are easily led astray by irrelevant retrieved text. Furthermore, prior RL-based prompt optimization primarily optimizes task instructions (e.g., "classify sentiment"), which share a small, static search space. In contrast, personal context components change with every user and document, creating an enormous search space where standard RL exploration fails.

Methods

The authors tackle prompt optimization for context-dependent elements—namely the summary and synthesis (keywords and writing style)—using a two-stage training paradigm:

  1. Writing Style Synthesis: The authors prompt an LLM to distill an author's historical documents into stylistic traits (e.g., "thorough", "engaging"), complementing keyword synthesis.
  2. Supervised Warm-up (SL): To constrain the RL search space, the authors generate synthetic training labels by randomly shuffling and pruning sentences, keywords, and style phrases (generating 65 prompt variants per document). The variant yielding the highest generation BLEU from the frozen LLM serves as the pseudo-ground-truth label to fine-tune a T5 sequence-to-sequence rewriter.
  3. Reinforcement Learning (RL): The supervised rewriter is further optimized via Proximal Policy Optimization (PPO). The action space spans the full vocabulary—allowing the rewriter to add new words, repeat terms, or delete elements—while using the frozen generator’s downstream text BLEU score as the reward signal.

During inference, only the lightweight rewriter and frozen generator run end-to-end.

Datasets

The authors evaluate on three representative domains partitioned by user (85% train, 5% validation, 10% test):

  • Avocado Research Email Collection: Long-form personal corporate email generation.
  • Amazon Reviews (Books): Medium-length product reviews with narrower topical scope.
  • Reddit Comments: Short, informal social discussions.

Key Insights and Ideas

Inspection of prompts produced by the SL-RL rewriter revealed surprising, interpretable patterns:

  • Drop the summary entirely: The rewriter consistently deleted the entire historical document summary. Extractive summaries often include off-topic sentences that confuse frozen LLMs, whereas fine-tuned models can selectively attend to useful tokens.
  • Repeat for emphasis: The rewriter learned to duplicate high-priority keywords (e.g., repeating "dragon" in a fantasy review prompt) to encourage the frozen generator to center its output around those themes.
  • Order follows chronology: Keywords were reordered from order-of-importance to their predicted narrative appearance order in the final output text.
  • Task-specific style cues: For long-form tasks (emails), the rewriter learned to inject uniform instructions like "the author is thorough" to steer the model toward detailed completions. For book reviews, it stripped explicit style descriptions completely, letting retrieved exemplar reviews implicitly define the tone.

Key Results

Using T5-11B as the rewriter and PaLM 2 as the frozen generator, the combined RewriterSlRl substantially outperformed original prompts and single-paradigm baselines:

  • Avocado Email: BLEU increased from 9.59 (Original) to 13.18 (RewriterSlRl), with ROUGE-1 rising from 30.04 to 33.74.
  • Amazon Reviews: BLEU jumped from 5.00 to 13.07, and ROUGE-L improved from 16.32 to 25.16.
  • Reddit Comments: BLEU rose from 13.13 to 20.59, with ROUGE-1 climbing from 31.92 to 40.28.
  • Component Synergy: Training with RL alone produced unreadable gibberish (BLEU dropped to 3.53 on Amazon), while SL alone offered marginal gains. Only the chained SL-to-RL pipeline unlocked significant improvements.
  • Actionable Heuristics: Manually applying the rewriter's discovered rules (e.g., dropping the summary and retaining only keywords and style) improved standard prompts without requiring any automated rewriter at runtime.

Limitations and Future Work

  • Fixed Prompt Components: The framework only rewrites the summary and synthesis sections; task instructions and retrieved historical in-context examples remain static.
  • Training Expense: Generating pseudo-labels for the initial SL stage requires querying the target LLM dozens of times per training instance, which is computationally expensive during offline preparation.
  • Metric Limitations: RL rewards were optimized solely against BLEU word overlap, which may not capture subtle factual consistency or nuanced human preferences.

Summary updated

Abstract

Facilitated by large language models (LLMs), personalized text generation has become a rapidly growing research direction. Most existing studies focus on designing specialized models for a particular domain, or they require fine-tuning the LLMs to generate personalized text. We consider a typical scenario in which the large language model, which generates personalized output, is frozen and can only be accessed through APIs. Under this constraint, all one can do is to improve the input text (i.e., text prompts) sent to the LLM, a procedure that is usually done manually. In this paper, we propose a novel method to automatically revise prompts for personalized text generation. The proposed method takes the initial prompts generated by a state-of-the-art, multistage framework for personalized generation and rewrites a few critical components that summarize and synthesize the personal context. The prompt rewriter employs a training paradigm that chains together supervised learning (SL) and reinforcement learning (RL), where SL reduces the search space of RL and RL facilitates end-to-end training of the rewriter. Using datasets from three representative domains, we demonstrate that the rewritten prompts outperform both the original prompts and the prompts optimized via supervised learning or reinforcement learning alone. In-depth analysis of the rewritten prompts shows that they are not only human readable, but also able to guide manual revision of prompts when there is limited resource to employ reinforcement learning to train the prompt rewriter, or when it is costly to deploy an automatic prompt rewriter for inference.
BibTeX
@inproceedings{li2024learning,
  author = {Cheng Li and Mingyang Zhang and Qiaozhu Mei and Weize Kong and Michael Bendersky},
  title = {{Learning to Rewrite Prompts for Personalized Text Generation}},
  booktitle = {The Web Conference},
  year = {2024},
  doi = {10.1145/3589334.3645408},
}
Figure