The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving

Max Ruiz Luyten, Mihaela van der Schaar

TL;DR

This paper introduces Distributional Creative Reasoning (DCR), a novel framework for training Large Language Models (LLMs) that addresses the "creative collapse" problem. DCR ensures LLMs remain both accurate and creative by optimizing a diversity-driven objective, preventing them from converging to a narrow set of redundant solutions.

What Problem Does This Paper Address?

Modern LLM training pipelines, especially those using reinforcement learning (RL) for mathematical reasoning, often optimize for correctness. While this improves accuracy, it leads to "creative collapse," where the model's output becomes repetitive and lacks diversity. This is problematic because creative problem-solving, which relies on a diverse set of strategies, is crucial for generalization to out-of-distribution tasks and for genuine discovery. The paper aims to understand why this collapse happens and how to prevent it without sacrificing accuracy.

Methods

The core contribution is the Distributional Creative Reasoning (DCR) framework. DCR recasts LLM training as a gradient flow on the simplex of probability distributions over solution traces. This allows for a principled analysis of diversity decay. The DCR objective includes:

  • Utility: Rewards correct outputs.
  • Diversity Energy: A novel term that promotes both breadth (Shannon entropy) and semantic distinctiveness (using a creativity kernel).
  • KL-Divergence: Acts as a regularizer to maintain stability.

The paper provides a theoretical analysis of how common training objectives (STaR, GRPO, DPO) lead to different modes of diversity decay. It then proves that the DCR objective, with its diversity energy, guarantees convergence to a stable and diverse policy, effectively preventing collapse. The framework also offers practical guidance on designing the creativity kernel and tuning hyperparameters.

Datasets

Not discussed in the paper. The experiments use a synthetic "trace universe" with 12 traces (8 correct, 4 incorrect) partitioned into 3 semantic clusters.

Key Insights and Ideas

  • Creative Collapse is Systemic: The paper demonstrates that optimizing solely for correctness in LLM training inherently leads to a loss of diversity, not as a bug, but as a feature of certain optimization objectives.
  • Diversity is Not Just Entropy: While entropy promotes a broad distribution, it's insufficient for true creativity. A "creativity kernel" is needed to actively penalize semantically similar traces, encouraging distinct problem-solving strategies.
  • DCR Offers a Principled Solution: By framing training as a gradient flow on the distribution of traces and incorporating a structured diversity term, DCR provides a theoretically grounded way to achieve both correctness and creativity.
  • Algorithm-Specific Decay: Different RL algorithms (STaR, GRPO, DPO) exhibit distinct patterns of diversity decay, which DCR's framework can predict and counteract.

Key Results

  • Diversity Decay Theorem: Formally characterizes the distinct modes of diversity decay for STaR ("winner-takes-all"), GRPO ("neutral drift"), and DPO ("homogenization") under scalar objectives.
  • Guaranteed Convergence: The DCR objective is proven to guarantee convergence to a unique, stable, and diverse interior equilibrium policy.
  • Practical Design Levers: The framework provides actionable recipes for designing creativity kernels and tuning hyperparameters to achieve desired levels of diversity and correctness.
  • Empirical Validation: Experiments show that DCR successfully maintains diversity and safety margins while achieving high performance, outperforming entropy-only or ungated approaches.

Limitations and Future Work

  • Computational Cost: The kernel-driven diversity term can be computationally intensive (O(B²) per step for SGD), though practical strategies are discussed.
  • Kernel Design: While DCR provides a principled approach, designing effective semantic kernels (ksem) for real-world tasks remains an area for further research.
  • Generalization to Diverse Tasks: The current work focuses on mathematical reasoning; extending DCR to other creative tasks (e.g., story generation, code generation) would be valuable.
  • Theoretical Guarantees for Learned Kernels: Further theoretical analysis of learned semantic kernels and their impact on diversity would be beneficial.

Summary updated

Abstract

State-of-the-art post-training pipelines for reasoning LLMs rely on bootstrapped reasoning loops: they sample many traces, score them, and reinforce the highest-scoring ones, typically by correctness. This can improve accuracy while still collapsing the distribution inside the correct set onto a narrow family of redundant strategies, reducing creative problem-solving. To diagnose this failure mode, we introduce Distributional Creative Reasoning (DCR), a variational framework that casts training as gradient flow on the simplex of reasoning traces. The framework yields three core results. First, a diversity-decay analysis shows that STaR-style rejection fine-tuning and exact mean-field GRPO amplify whichever correct trace is already larger, while DPO regresses pairwise correct-trace ratios toward the reference ratios. Second, it explains why entropy and KL can slow or tether collapse but do not reward semantically distinct correct strategies for being distinct, and how a creativity kernel supplies the missing relational term. Third, under mild conditions, the resulting dynamics converge to a unique, stable, and diverse equilibrium, yielding practical guidance for kernel and hyperparameter design. DCR thus offers a principled route to training reasoning LLMs that remain both correct and creative.
BibTeX
@inproceedings{luyten2026reasoning,
  author = {Max Ruiz Luyten and Mihaela van der Schaar},
  title = {{The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=2xNpScK097},
}
Figure