TL;DR
This paper introduces Distributional Creative Reasoning (DCR), a novel framework for training Large Language Models (LLMs) that addresses the "creative collapse" problem. DCR ensures LLMs remain both accurate and creative by optimizing a diversity-driven objective, preventing them from converging to a narrow set of redundant solutions.
What Problem Does This Paper Address?
Modern LLM training pipelines, especially those using reinforcement learning (RL) for mathematical reasoning, often optimize for correctness. While this improves accuracy, it leads to "creative collapse," where the model's output becomes repetitive and lacks diversity. This is problematic because creative problem-solving, which relies on a diverse set of strategies, is crucial for generalization to out-of-distribution tasks and for genuine discovery. The paper aims to understand why this collapse happens and how to prevent it without sacrificing accuracy.
Methods
The core contribution is the Distributional Creative Reasoning (DCR) framework. DCR recasts LLM training as a gradient flow on the simplex of probability distributions over solution traces. This allows for a principled analysis of diversity decay. The DCR objective includes:
- Utility: Rewards correct outputs.
- Diversity Energy: A novel term that promotes both breadth (Shannon entropy) and semantic distinctiveness (using a creativity kernel).
- KL-Divergence: Acts as a regularizer to maintain stability.
The paper provides a theoretical analysis of how common training objectives (STaR, GRPO, DPO) lead to different modes of diversity decay. It then proves that the DCR objective, with its diversity energy, guarantees convergence to a stable and diverse policy, effectively preventing collapse. The framework also offers practical guidance on designing the creativity kernel and tuning hyperparameters.
Datasets
Not discussed in the paper. The experiments use a synthetic "trace universe" with 12 traces (8 correct, 4 incorrect) partitioned into 3 semantic clusters.
Key Insights and Ideas
- Creative Collapse is Systemic: The paper demonstrates that optimizing solely for correctness in LLM training inherently leads to a loss of diversity, not as a bug, but as a feature of certain optimization objectives.
- Diversity is Not Just Entropy: While entropy promotes a broad distribution, it's insufficient for true creativity. A "creativity kernel" is needed to actively penalize semantically similar traces, encouraging distinct problem-solving strategies.
- DCR Offers a Principled Solution: By framing training as a gradient flow on the distribution of traces and incorporating a structured diversity term, DCR provides a theoretically grounded way to achieve both correctness and creativity.
- Algorithm-Specific Decay: Different RL algorithms (STaR, GRPO, DPO) exhibit distinct patterns of diversity decay, which DCR's framework can predict and counteract.
Key Results
- Diversity Decay Theorem: Formally characterizes the distinct modes of diversity decay for STaR ("winner-takes-all"), GRPO ("neutral drift"), and DPO ("homogenization") under scalar objectives.
- Guaranteed Convergence: The DCR objective is proven to guarantee convergence to a unique, stable, and diverse interior equilibrium policy.
- Practical Design Levers: The framework provides actionable recipes for designing creativity kernels and tuning hyperparameters to achieve desired levels of diversity and correctness.
- Empirical Validation: Experiments show that DCR successfully maintains diversity and safety margins while achieving high performance, outperforming entropy-only or ungated approaches.
Limitations and Future Work
- Computational Cost: The kernel-driven diversity term can be computationally intensive (O(B²) per step for SGD), though practical strategies are discussed.
- Kernel Design: While DCR provides a principled approach, designing effective semantic kernels (ksem) for real-world tasks remains an area for further research.
- Generalization to Diverse Tasks: The current work focuses on mathematical reasoning; extending DCR to other creative tasks (e.g., story generation, code generation) would be valuable.
- Theoretical Guarantees for Learned Kernels: Further theoretical analysis of learned semantic kernels and their impact on diversity would be beneficial.
Summary updated