TL;DR
Researchers have introduced Creative Preference Optimization (CRPO), a modular alignment method that allows Large Language Models (LLMs) to directly optimize for novelty, diversity, and surprise. By leveraging MUCE, a massive new dataset of 200,000+ human-rated creative responses, the authors produced models that significantly outperform GPT-4o and Claude-3.7 on creativity benchmarks without sacrificing output quality.
What Problem Does This Paper Address?
While LLMs are incredibly capable, they suffer from a "regression to the mean." Standard alignment techniques like RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) tend to encourage models to produce "safe," helpful, but ultimately unoriginal content—a phenomenon often called "mode collapse."
True creativity is multifaceted, requiring novelty (originality), diversity (variety), surprise (unexpectedness), and quality (utility). Most existing research focuses on only one of these (usually diversity) or targets a single task (like story writing). There has been a lack of both a unified optimization framework and a large-scale, psychologically valid dataset to train models to think "out of the box" across diverse domains.
Methods
The core contribution is Creative Preference Optimization (CRPO). It builds on the standard DPO framework but modifies the loss function to inject specific signals for different dimensions of creativity.
The authors define a modular objective where the standard DPO loss is multiplied by a weighted combination of four unsupervised metrics:
- Diversity ($\delta^w$): Measured as the average pairwise semantic distance between a set of preferred responses.
- Novelty ($\nu^w$): Calculated using "divergent semantic integration"—the absolute difference between the semantic distance of words in a response versus a reference corpus.
- Surprise ($\xi^w$): Operationalized as the perplexity of the response under a reference model (Gemma-2-27B).
- Quality ($\gamma^w$): Derived from an external reward model (Skywork-Reward).
By adjusting the "injection weights" ($\lambda$), researchers can tune a model to be specifically more surprising, more diverse, or a balanced "creative" mix.
Datasets
- MUCE (Multitask Creativity Evaluation): A major contribution of the paper. It contains over 200,000 human-generated responses and ratings across 30+ psychological creativity assessments (e.g., the Alternative Uses Task, Metaphor generation, and Hypothesis generation). Crucially, 43% of this data was previously unreleased, making it a "clean" benchmark for LLMs.
- NOVELTYBENCH: An external, recently introduced benchmark used to test the generalizability of the CRPO-trained models.
Key Insights and Ideas
- The Creativity-Quality Trade-off: The paper quantifies the tension between being "creative" and being "useful." Injecting high levels of surprise or novelty can lead to a slight drop in quality scores, but the modular nature of CRPO allows developers to find the "sweet spot" for specific applications.
- DPO is a Creativity Killer: The authors found that vanilla DPO (without creativity signals) often produces less diverse and less novel outputs than even simple supervised fine-tuning (SFT).
- Novelty $\neq$ Surprise: Interestingly, increasing novelty injection often leads to higher surprise, but the inverse isn't always true. More surprise can sometimes cause a drop in novelty, suggesting these are distinct cognitive dimensions that need separate optimization.
- Psychological Grounding: By using tasks from established psychological literature rather than just "web-scraped" preferences, the models learn a more robust, human-like definition of creativity.
Key Results
- Outperforming the Giants: CRPO-enhanced versions of Llama-3.1-8B and Mistral-7B outperformed GPT-4o, Claude-3.7-Sonnet, and Gemini-2.0-Flash across automated metrics for novelty, diversity, and surprise.
- Human Preference: In blind pairwise comparisons, human raters consistently judged CRPO model outputs as more creative than those from GPT-4o and standard DPO baselines.
- State-of-the-Art: CRPO models set new records on the NOVELTYBENCH leaderboard for novelty metrics.
- Quantitative Leap: In the Alternative Uses Task (AUT), the CRPO-sur model achieved a surprise score of 590.65, compared to GPT-4o’s 42.63, while maintaining a competitive quality score.
Limitations and Future Work
- Safety and "Malevolent Creativity": The authors acknowledge a significant ethical risk: a more creative model is also better at generating "creative" ways to do harm (e.g., novel sabotage strategies). Future work must balance creativity with robust safety guardrails.
- Scaling: The experiments were limited to 7B and 8B parameter models. Testing whether these creative signals scale to 70B+ models is a necessary next step.
- Language Scope: While MUCE contains multilingual data, this paper focused primarily on English.
- Subjectivity: Creativity remains inherently subjective. While the metrics used are grounded in research, they may not capture the full "spark" of human genius.
Summary updated