Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems

Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin, Grace Guiling WangAAAI 2025 · DOI

Introduction

Traditional AI evaluation in mathematics has primarily focused on accuracy—the ability of a model to reach the correct final answer. However, true mathematical discovery requires more than just correctness; it demands creativity. Researchers have introduced CREATIVEMATH, a benchmark designed to evaluate whether Large Language Models (LLMs) can generate novel solutions to complex problems. By defining creativity as the production of solutions that are both useful (correct) and novel (distinct from known methods), this study shifts the focus from "what" the answer is to "how" it is derived.

The CREATIVEMATH Framework

To assess creativity, the researchers developed a multi-stage pipeline that prompts LLMs to think "outside the box."

  1. Novel Solution Generation: An LLM is provided with a math problem and a set of $k$ existing human solutions. It is then tasked with generating a new solution that differs fundamentally in methodology, intermediate steps, or underlying assumptions.
  2. Evaluation Pipeline: Judging novelty is more complex than checking a numerical answer. The study employs a "committee" of three state-of-the-art models—GPT-4, Claude 3.5 Sonnet, and Gemini 1.5 Pro—to evaluate the generated solutions.
  3. Correctness Check: Only solutions that are mathematically sound proceed to novelty assessment.
  4. Coarse-Grained Novelty: The solution is compared against the $k$ provided reference solutions.
  5. Fine-Grained Novelty: The solution is compared against all known human solutions in the dataset to ensure true originality.
The CREATIVEMATH evaluation pipeline, illustrating the generation of new solutions and the multi-model consensus process for judging correctness and novelty.

Dataset and Scope

The CREATIVEMATH dataset is a curated collection from the Art of Problem Solving (AoPS) platform, comprising 6,469 problems and over 14,000 solutions. The problems span a wide range of difficulties, from middle school competitions like the AMC 8 to elite international Olympiads such as the IMO.

Distribution of math problems across categories and competitions in the CREATIVEMATH dataset.

The dataset is categorized into fields such as Algebra, Geometry, Number Theory, and Probability. This diversity allows for a granular analysis of where LLMs exhibit the most creativity. For the primary experiments, a subset of 400 problems and 605 solution samples was used to rigorously test model performance.

Key Insights: The Creativity Trade-off

The study revealed several critical insights into the nature of LLM "thinking":

  • The Inverse Relationship of Accuracy and Novelty: A significant "Creativity Trade-off" was observed. As the number of reference solutions ($k$) provided in the prompt increases, the models' accuracy improves because they have more "hints." However, their ability to produce novel solutions decreases significantly, as they tend to mimic the provided patterns rather than innovating.
  • Difficulty as a Catalyst for Innovation: Counterintuitively, harder problems (e.g., USAMO or IMO level) often resulted in a higher ratio of novel solutions among the correct ones. When standard paths are difficult to find or execute, models are seemingly forced to explore more unconventional methodologies.
  • Model Performance: Gemini 1.5 Pro emerged as the most creative model, achieving a Novelty Ratio ($N$) of 66.94% and a near-perfect Novelty-to-Correctness ratio ($N/C$) of 95.75%. Among open-source models, Llama-3-70B showed strong creative potential, often outperforming larger closed-source models like GPT-4o in novelty metrics.

Conclusion and Future Directions

While the CREATIVEMATH benchmark provides a robust framework for assessing mathematical creativity, it is not without limitations. The reliance on LLMs as judges, while efficient, still lacks the nuance of human expert validation. Furthermore, the risk of data leakage from public competition problems remains a challenge for all LLM evaluations.

Future work will focus on integrating "associative thinking" techniques to help models bridge disparate mathematical concepts and automating the cleaning of model outputs to ensure unbiased evaluation. By moving beyond simple correctness, this research paves the way for AI that can truly partner with humans in the process of scientific and mathematical discovery.

Summary updated

BibTeX
@inproceedings{ye2025assessing,
  author = {Junyi Ye and Jingyi Gu and Xinyun Zhao and Wenpeng Yin and Grace Guiling Wang},
  title = {{Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34760},
}
Figure