TL;DR
This paper introduces Creative Priming (CP), a novel framework that escapes the "creative paradox"—the tendency for AI models to become less creative as they are trained to mimic past creative successes. By decoupling the creative process (generating a strategy) from the product (the final response), the authors demonstrate significant gains in novelty and diversity that transfer even to frontier models like GPT-5 and Claude.
What Problem Does This Paper Address?
Creativity is a "vexing paradox" for machine learning. Typically, we train models by reinforcing successful past behaviors. However, creativity is defined by novelty: as soon as a creative act is recorded and used for training, it is no longer novel. Reinforcing specific creative outputs makes the model more likely to repeat them, which is the antithesis of creativity.
The authors argue that current models suffer from "homogeneity," where they collapse onto a handful of common ideas. While existing methods try to reward "diversity," they often trade off quality for variety. The core challenge is learning the generative principles of creativity without just memorizing creative artifacts.
Methods
The authors propose Creative Priming (CP), which splits the generation task into two distinct stages:
- The Priming Model ($\pi_{CP}$): A model trained to generate a "primer"—a short text containing meta-level creative guidance (e.g., "Focus on a single object that shouldn't exist in this era").
- The Response Model ($\pi_{resp}$): A frozen, high-quality model that receives the primer as additional context to generate the final answer.
The priming model is trained using Group-Relative Policy Optimization (GRPO). Crucially, it is rewarded based on the creativity of the responses it induces in the frozen model, rather than its own output. The reward function balances three pillars:
- Quality: Is the response coherent and helpful?
- Deviation: How far does the response depart from a standard "reference" set of answers?
- Diversity: How different are the responses from each other?
By keeping the response model frozen, the system avoids the creative paradox: it reinforces the process of being creative without ever telling the response model to repeat a specific "creative" answer.
Datasets
- Training: A filtered subset of Infinite-Chats (derived from WildChat), selected for high open-endedness, complexity, and creative potential (approx. 21,000 prompts).
- Evaluation: NoveltyBench, a benchmark of 100 real-world open-ended prompts designed to elicit creative thinking. The authors extended this with custom LLM-as-a-judge prompts to measure nuanced quality.
Key Insights and Ideas
- Decoupling is Key: When you train a model directly for creativity, it often sacrifices quality to achieve novelty (becoming terse or nonsensical). By using a primer, the response model can stay in its "high-quality" regime while the primer pushes it into "novel" regions of its latent space.
- Diversity vs. Deviation: The authors make a vital distinction. Diversity is the spread within a group of new responses. Deviation is how far those responses are from what the model usually says. A creative system needs both.
- The Creativity Ceiling: The paper finds that simply scaling up standard reinforcement learning (like DGRPO) eventually hits a wall where quality and diversity start to decline. Creative Priming appears to bypass this ceiling.
- Modular Creativity: Because the priming model learns general "thinking strategies," it is highly transferable. A primer model trained on a small 8B model can successfully "prime" much larger frontier models it has never seen before.
Key Results
- Performance Boost: On NoveltyBench, CP produced a 39–200% relative increase in responses that were simultaneously novel, high-quality, and semantically distinct compared to direct training methods.
- Zero-Shot Transfer: When paired with frontier API models (Claude 3.5 Sonnet, GPT-5, etc.), the CP primer increased the number of distinct responses by 4.5–10× and diversity by 2.5–4.6×, all while maintaining or slightly improving quality.
- Qualitative Shift: Qualitative analysis (Figure 3) shows that while baseline models often change their style (e.g., becoming bulleted lists) to seem different, CP changes the content (inventing novel, concrete images) while maintaining a natural prose style.
Limitations and Future Work
- Inference Overhead: The two-stage process (generating a primer, then a response) increases latency and compute costs.
- Capability Bounds: A primer can only elicit what a frozen model is already capable of; it cannot "teach" the model new facts or skills it doesn't possess.
- Reward Signal Dependency: The system relies on LLM-as-a-judge and embedding models for rewards, which may have their own biases or blind spots regarding what constitutes "creativity."
- Future Directions: The authors suggest exploring "internalized" priming (e.g., via Chain-of-Thought) and using non-stationary reference sets that evolve as the model generates more ideas.
Summary updated