AI-Assisted Generation of Difficult Math Questions

Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Nan Rosemary Ke, Michael Curtis Mozer, Yoshua Bengio, Sanjeev Arora, Anirudh GoyalSubmission to ICLR 2025 · Rejected

TL;DR

As public math benchmarks like MATH become saturated and memorized by frontier language models, the authors propose an AI-assisted, human-in-the-loop pipeline that generates difficult, novel math questions by deliberately composing two distinct mathematical skills. The resulting dataset, MATH$^2$, severely challenges current models, revealing a striking quadratic scaling property: a model's accuracy on MATH$^2$ roughly equals its accuracy on MATH squared ($Y \approx X^2$).

What Problem Does This Paper Address?

Frontier large language models (LLMs) are running out of high-quality, un-contaminated evaluation data. Current math benchmarks (such as GSM8K and MATH) suffer from benchmark saturation—partly due to legitimate capability gains, but also due to supervised fine-tuning (SFT) on synthetic variations of test sets, causing models to overfit rather than acquire genuine reasoning skills.

Creating genuinely challenging questions exclusively with human experts is slow and expensive. Conversely, relying purely on LLMs to generate synthetic problems often yields questions that are trivial, logically flawed, or simply tedious (conflating mathematical difficulty with grueling arithmetic calculations).

Methods

The authors introduce a structured five-step AI pipeline combined with expert human refinement:

  1. Skill Extraction & Pairing: The authors extract 114 fine-grained mathematical skills from the MATH benchmark (spanning algebra, geometry, number theory, etc.). To force out-of-distribution compositional reasoning, the pipeline samples pairs of non-overlapping skills across different topics (e.g., combining modular arithmetic with geometry).
  2. Skill Pair Validation: An LLM verifies that the two sampled skills are genuinely distinct and not redundant.
  3. Question Generation: An LLM generates a question and initial solution requiring both skills, guided by multi-turn human-AI demonstration dialogues.
  4. Adversarial Solution Attempt: The LLM attempts to solve the generated question with an explicit "defeatist" objective—actively searching for ambiguities, missing information, self-contradictions, or computational intractability.
  5. Multi-Criteria Validation & Re-solving: The problem is vetted against a 7-point rubric (including single/exact answer requirements, dual-skill necessity, and tractability) using majority voting (maj@4). It is then re-solved using self-consistency; problems yielding multiple conflicting answers are discarded.
  6. Human-in-the-Loop Refinement: Human mathematicians review candidate questions to patch subtle reasoning gaps, eliminate tedious enumeration, and heighten conceptual depth.

Datasets

  • MATH Dataset (Hendrycks et al., 2021): Used as the seed repository for extracting skills and as the baseline evaluation suite.
  • MATH$^2$: The paper's newly introduced evaluation set containing 210 human-verified, multi-skill math questions generated via GPT-4 Turbo, GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro.

Key Insights and Ideas

  • The "Quadratic Law" of Composition ($Y \approx X^2$): When plotting accuracy on MATH$^2$ ($Y$) against accuracy on MATH ($X$), the data follows a near-perfect quadratic relationship: $Y \approx X^2$. Heuristically, if solving a problem requires independently and correctly executing two distinct mathematical skills ($s_i \times s_j$), a model's overall success rate naturally drops to the square of its single-skill competency.
  • Specialized Synthetic Models Overfit: Open-source models fine-tuned heavily on MATH-like synthetic data (e.g., MetaMath, MAmmoTH) collapsed significantly on MATH$^2$, showing relative performance drops of up to 93.9%.
  • Supercharging In-Context Learning: Questions that fuse multiple concepts make superior few-shot exemplars. Prompting models on standard MATH problems using MATH$^2$ questions as in-context demonstrations consistently outperformed standard few-shot baselines.

Key Results

  • Across-the-Board Performance Drops:
  • GPT-4 Omni: 77.21% on MATH $\rightarrow$ 64.29% on MATH$^2$ (16.73% relative drop).
  • Claude 3.5 Sonnet: 73.54% on MATH $\rightarrow$ 46.15% on MATH$^2$ (37.24% relative drop).
  • Llama-3.1-70B-Instruct: 67.40% on MATH $\rightarrow$ 50.48% on MATH$^2$ (25.10% relative drop).
  • MAmmoTH-7B: 7.90% on MATH $\rightarrow$ 0.48% on MATH$^2$ (93.92% relative drop).
  • Exemplar Effectiveness: Using MATH$^2$ questions as 4-shot in-context exemplars raised Llama-3.1-70B's accuracy on MATH from 58.15% (standard baseline) to 65.59% (a 12.79% relative improvement).
  • Human Modification Statistics: Out of 210 questions, 66.19% required human adjustments to either the problem statement or solution—underscoring that autonomous generation still cannot bypass human oversight for high-level math.

Limitations and Future Work

  • High Cost and Scalability Limits: The generation pipeline required expensive API calls across multiple frontier models (~$3,919 total) and substantial expert human time, constraining MATH$^2$ to 210 problems.
  • Model Failure Modes: Models still struggled with "finding a needle in a haystack" during long validation prompts, generating deceitful/hallucinated solutions, and leaning toward tedious enumeration rather than elegant mathematical logic.
  • Future Directions: Incorporating open-weights models, utilizing autoformalization/code-executors to verify solutions programmatically, closing the loop with fine-tuning on verified problems, and extending skill-compositional generation beyond mathematics.

Summary updated

Abstract

Current LLM training positions mathematical reasoning as a core capability. With publicly available sources fully tapped, there is an unmet demand for diverse and challenging mathematics questions. Relying solely on human experts is both time-consuming and costly, while LLM-generated questions often lack the requisite diversity and difficulty. We present a design framework that combines the strengths of LLMs with a human-in-the-loop approach to generate a diverse array of challenging math questions. Initially, leveraging LLM metacognition skills [Didolkar et al., 2024], a strong LLM is used to extract core "skills'' from existing math datasets. These skills serve as the basis for generating novel and difficult questions by prompting the LLM with random pairs of core skills that must be utilized in the question. The use of two very different skills within each question makes finding such questions an ``out of distribution'' task for both LLMs and humans. Our pipeline employs LLMs to iteratively generate and refine questions and solutions through multi-turn prompting. Human annotators then verify and further refine the questions, with their efficiency enhanced via further LLM interactions. Applying this pipeline on skills extracted from MATH dataset [Hendrycks et al., 2024] resulted in **MATH$^2$** - a dataset of higher quality math questions, as evidenced by: (a) Lower performance of all models on MATH$^2$ than on MATH (b) Higher performance on MATH when using MATH$^2$ questions as in-context examples. Although focused on mathematics, our methodology seems applicable to other domains requiring structured reasoning, and potentially as a component of *scalable oversight*. Also of interest is a striking relationship observed between models' performance on the new dataset: the success rate on MATH$^2$ is the square on MATH. This suggests that successfully solving the question in MATH$^2$ requires a nontrivial combination of two distinct math skills.
BibTeX
@inproceedings{shah2024ai,
  author = {Vedant Shah and Dingli Yu and Kaifeng Lyu and Simon Park and Nan Rosemary Ke and Michael Curtis Mozer and Yoshua Bengio and Sanjeev Arora and Anirudh Goyal},
  title = {{AI-Assisted Generation of Difficult Math Questions}},
  booktitle = {MATH-AI @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=6pUdfsJmd1},
}
Figure