DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing

Qian Cao, Yahui Liu, Wei Bi, Yi Zhao, Ruihua Song, Xiting Wang, Ruiming Tang, Guorui Zhou, Han Li

TL;DR

Large language models (LLMs) trained with reinforcement learning (RL) often lose their creative diversity. This paper introduces DPWriter, a novel RL framework that uses a structured planning phase within a Chain-of-Thought (CoT) to guide diverse exploration, significantly boosting output variety in creative writing without sacrificing quality.

What Problem Does This Paper Address?

Creative writing with LLMs is a prime example of an open-ended task where diversity is crucial. However, a common issue arises when LLMs are fine-tuned using reinforcement learning (RL), particularly with methods like RLHF (Reinforcement Learning from Human Feedback). This process, while improving alignment with human preferences and overall performance, tends to reduce the variety of generated outputs. Existing RL methods often lack explicit mechanisms to encourage diverse exploration, instead prioritizing efficiency and quality. This leads to LLMs producing repetitive or predictable content, hindering their utility for creative endeavors. The paper aims to solve this "diversity collapse" problem in RL-enhanced LLMs for creative writing.

Methods

The core of DPWriter is a semi-structured long Chain-of-Thought (CoT). Instead of a free-form reasoning process, DPWriter decomposes generation into distinct stages:

  1. Global Planning Phase: Before generating the reasoning chain, the model first creates a high-level plan (p) for the entire response. This plan is derived from the instruction and guides subsequent steps.
  2. Plan-Consistent CoT Generation: A reasoning chain (c') is then generated, conditioned on both the instruction and the plan, ensuring coherence.
  3. Response Generation: Finally, the response (y) is generated based on the instruction, plan, and reasoning chain.

To achieve diverse exploration during RL training, DPWriter introduces two key mechanisms:

  • Diverse Planning Branching (DPB): At specific "planning segments" within the CoT, the model strategically branches out multiple diverse plans. These candidate plans are then evaluated for diversity using n-gram-based and semantic metrics, and the most diverse ones are selected to proceed. This allows for controlled exploration of different creative trajectories.
  • Group-Aware Diversity Reward: A novel reward function is introduced that not only considers the quality of a response but also its contribution to the diversity of a group of generated responses. This encourages the model to produce outputs that are unique within a batch, further promoting variety. This reward is only applied if the response meets a minimum quality threshold, ensuring that diversity doesn't come at the expense of coherence.

The framework builds upon the Group Relative Policy Optimization (GRPO) algorithm.

Datasets

The paper utilizes several existing creative writing datasets for training and evaluation:

  • DeepWriting
  • Writing-Prompts
  • CreateSet
  • COIG-Writer

For datasets that only contain instructions and responses, they generated long CoTs using GPT-4.1. A curated dataset of approximately 43K writing instructions with semi-structured long CoT and high-quality responses was constructed for training.

Key Insights and Ideas

  • Structured Planning is Key to Diversity: By introducing an explicit planning phase before reasoning, DPWriter gains a crucial control point to steer the generation process towards diversity. This contrasts with unstructured CoT methods where diversity is harder to influence.
  • Branching at the Planning Stage is More Controllable: Unlike prior work that branched at high-entropy tokens, branching at the planning stage allows for more deliberate and guided exploration of diverse creative directions.
  • Diversity Reward Needs Quality Guardrail: The diversity contribution reward is most effective when paired with a quality threshold. This prevents the model from generating diverse but nonsensical outputs, ensuring that diversity is pursued within the bounds of good writing.
  • Synergy Between Branching and Reward: The paper demonstrates that the DPB method and the diversity reward work synergistically, with the combined approach yielding significantly better diversity scores than either component alone.

Key Results

  • Significant Diversity Improvement: DPWriter consistently outperforms baselines on diversity metrics (e.g., embedding-based diversity, EAD, Distinct) across multiple benchmarks (WritingBench, Creative Writing v3, ArenaHard v2.0). For instance, on WritingBench with the Qwen3-4B backbone, DPWriter achieved a 15% improvement in embedding-based diversity and a 9.9% improvement in EAD compared to standard GRPO.
  • Maintained High Quality: Crucially, DPWriter achieves these diversity gains without sacrificing generation quality. It often matches or surpasses baselines in quality scores (e.g., highest overall score of 6.43 on WritingBench).
  • Ablation Studies Confirm Component Importance: Removing the branching strategy or the diversity reward significantly degrades performance on diversity metrics, highlighting their critical roles.
  • Case Studies Show Tangible Differences: Qualitative examples (Figure 5, Figure 8) illustrate how DPWriter generates more varied and distinct responses compared to baselines, which tend to produce similar outputs.

Limitations and Future Work

  • Computational Overhead: The reliance on semi-structured CoT and DPB might introduce additional computational overhead, potentially limiting scalability for extremely large models or datasets.
  • Quality-Diversity Trade-off: While the paper shows significant improvements in both, the inherent "seesaw" between quality and diversity might not be fully resolved, warranting further research.
  • Defining "Creativity": The paper focuses on diversity as a proxy for creativity. Further exploration is needed to understand if and how diversity can directly benefit other aspects of creativity beyond just variety.

Summary updated

Abstract

Reinforcement learning (RL)-based enhancement of large language models (LLMs) often leads to reduced output diversity, undermining their utility in open-ended tasks like creative writing. Current methods lack explicit mechanisms for guiding diverse exploration and instead prioritize optimization efficiency and performance over diversity. This paper proposes an RL framework structured around a semi-structured long Chain-of-Thought (CoT), in which the generation process is decomposed into explicitly planned intermediate steps. We introduce a Diverse Planning Branching method that strategically introduces divergence at the planning phase based on diversity variation, alongside a group-aware diversity reward to encourage distinct trajectories. Experimental results on creative writing benchmarks demonstrate that our approach significantly improves output diversity without compromising generation quality, consistently outperforming existing baselines.
BibTeX
@inproceedings{cao2026dpwriter,
  author = {Qian Cao and Yahui Liu and Wei Bi and Yi Zhao and Ruihua Song and Xiting Wang and Ruiming Tang and Guorui Zhou and Han Li},
  title = {{DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.647},
}

Loading related content...

Figure