TL;DR
When using frozen, API-only large language models (LLMs) for personalized text generation, static or noisy personal contexts degrade output quality. This paper introduces an automatic prompt-rewriting framework that chains supervised learning (SL) with reinforcement learning (RL) to dynamically optimize input-dependent context—specifically user summaries, style profiles, and keywords. By teaching a smaller model (T5) to rewrite prompts targeting a frozen generator (PaLM 2), the authors achieve dramatic generation quality improvements across emails, product reviews, and social media comments.
What Problem Does This Paper Address?
Personalized text generation aims to match an author's specific voice, topical interests, and stylistic habits. Recent frameworks (such as FtPersLlm) build rich prompt contexts containing retrieved past documents, extracted summaries, and keywords, but rely on fine-tuning the target LLM.
However, in most real-world applications, production LLMs are accessible only via black-box APIs (e.g., ChatGPT, PaLM 2) with frozen weights and non-differentiable generation metrics (like BLEU). While fine-tuned LLMs learn to ignore noisy context snippets, frozen LLMs are easily led astray by irrelevant retrieved text. Furthermore, prior RL-based prompt optimization primarily optimizes task instructions (e.g., "classify sentiment"), which share a small, static search space. In contrast, personal context components change with every user and document, creating an enormous search space where standard RL exploration fails.
Methods
The authors tackle prompt optimization for context-dependent elements—namely the summary and synthesis (keywords and writing style)—using a two-stage training paradigm:
- Writing Style Synthesis: The authors prompt an LLM to distill an author's historical documents into stylistic traits (e.g., "thorough", "engaging"), complementing keyword synthesis.
- Supervised Warm-up (SL): To constrain the RL search space, the authors generate synthetic training labels by randomly shuffling and pruning sentences, keywords, and style phrases (generating 65 prompt variants per document). The variant yielding the highest generation BLEU from the frozen LLM serves as the pseudo-ground-truth label to fine-tune a T5 sequence-to-sequence rewriter.
- Reinforcement Learning (RL): The supervised rewriter is further optimized via Proximal Policy Optimization (PPO). The action space spans the full vocabulary—allowing the rewriter to add new words, repeat terms, or delete elements—while using the frozen generator’s downstream text BLEU score as the reward signal.
During inference, only the lightweight rewriter and frozen generator run end-to-end.
Datasets
The authors evaluate on three representative domains partitioned by user (85% train, 5% validation, 10% test):
- Avocado Research Email Collection: Long-form personal corporate email generation.
- Amazon Reviews (Books): Medium-length product reviews with narrower topical scope.
- Reddit Comments: Short, informal social discussions.
Key Insights and Ideas
Inspection of prompts produced by the SL-RL rewriter revealed surprising, interpretable patterns:
- Drop the summary entirely: The rewriter consistently deleted the entire historical document summary. Extractive summaries often include off-topic sentences that confuse frozen LLMs, whereas fine-tuned models can selectively attend to useful tokens.
- Repeat for emphasis: The rewriter learned to duplicate high-priority keywords (e.g., repeating
"dragon"in a fantasy review prompt) to encourage the frozen generator to center its output around those themes. - Order follows chronology: Keywords were reordered from order-of-importance to their predicted narrative appearance order in the final output text.
- Task-specific style cues: For long-form tasks (emails), the rewriter learned to inject uniform instructions like
"the author is thorough"to steer the model toward detailed completions. For book reviews, it stripped explicit style descriptions completely, letting retrieved exemplar reviews implicitly define the tone.
Key Results
Using T5-11B as the rewriter and PaLM 2 as the frozen generator, the combined RewriterSlRl substantially outperformed original prompts and single-paradigm baselines:
- Avocado Email: BLEU increased from 9.59 (Original) to 13.18 (RewriterSlRl), with ROUGE-1 rising from 30.04 to 33.74.
- Amazon Reviews: BLEU jumped from 5.00 to 13.07, and ROUGE-L improved from 16.32 to 25.16.
- Reddit Comments: BLEU rose from 13.13 to 20.59, with ROUGE-1 climbing from 31.92 to 40.28.
- Component Synergy: Training with RL alone produced unreadable gibberish (BLEU dropped to 3.53 on Amazon), while SL alone offered marginal gains. Only the chained SL-to-RL pipeline unlocked significant improvements.
- Actionable Heuristics: Manually applying the rewriter's discovered rules (e.g., dropping the summary and retaining only keywords and style) improved standard prompts without requiring any automated rewriter at runtime.
Limitations and Future Work
- Fixed Prompt Components: The framework only rewrites the summary and synthesis sections; task instructions and retrieved historical in-context examples remain static.
- Training Expense: Generating pseudo-labels for the initial SL stage requires querying the target LLM dozens of times per training instance, which is computationally expensive during offline preparation.
- Metric Limitations: RL rewards were optimized solely against BLEU word overlap, which may not capture subtle factual consistency or nuanced human preferences.
Summary updated