RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation

Xinnuo Xu, Rachel Lawrence, Kshitij Dubey, Atharva Pandey, Risa Ueno, Fabian Falck, Aditya V. Nori, Rahul Sharma, Amit Sharma, Javier González

Introduction: Reasoning or Reciting?

The rapid advancement of Large Language Models (LLMs), exemplified by models like OpenAI’s o1 and DeepSeek-R1, has led to record-breaking scores on standard reasoning benchmarks. However, these results are often shadowed by the specter of data leakage. Since many benchmarks are public and static, models may simply be "reciting" patterns seen during pre-training rather than performing genuine logical reasoning. RE-IMAGINE addresses this by synthesizing novel, logically consistent variations of existing problems to test the limits of LLM cognition.

The RE-IMAGINE Framework

Inspired by Judea Pearl’s "Ladder of Causation," the authors propose a Ladder of Reasoning consisting of three levels:

  1. Level 1 (Observe): Solving original benchmark problems.
  2. Level 2 (Mutate): Solving variations with altered values or irrelevant information, maintaining the core logic.
  3. Level 3 (Imagine): Solving problems with new or counterfactual logic.

To scale this evaluation, RE-IMAGINE employs an automated pipeline that converts natural language (NL) questions into executable Python code, applies programmatic mutations, and then translates the result back into NL using a high-capacity model like GPT-4o.

The RE-IMAGINE pipeline: converting natural language questions to symbolic code, applying mutations, and translating back to natural language.

Because the intermediate representation is executable code, the ground truth for mutated problems is calculated automatically, ensuring high-fidelity evaluation without manual labeling.

Experimental Setup and Datasets

The framework was applied to four diverse benchmarks: GSM8K (math), CLadder (causal logic), CRUXEval (code execution), and Loop (software verification). A wide array of models was tested, including the GPT, Llama, and Phi families.

Key Insights: The Reasoning Gap

The primary finding is a consistent and sharp decline in performance as models move up the Ladder of Reasoning. Even high-performing models show a "Reasoning Gap" when faced with minor variations.

Performance degradation across various LLMs when subjected to Level 2 (Mutate) and Level 3 (Imagine) variations of the GSM8K benchmark.
  • Memorization Bias: A ~10% accuracy drop was observed across nearly all models when only numerical values were changed (SampleValues). This suggests models are anchored to specific patterns rather than the underlying arithmetic logic.
  • Brittle Logic: Level 3 mutations, which introduce new or contradictory premises, proved the most difficult. Models struggle to incorporate logic that deviates from the "standard" version of a problem.
  • Sensitivity to Noise: Combining mutations (e.g., changing values while adding irrelevant info) widened the performance gap, indicating that LLM reasoning is highly sensitive to environmental noise.

Conclusion

RE-IMAGINE reveals that current LLM performance on static benchmarks is an unreliable proxy for true reasoning capability. While models like GPT-o1 show improved robustness, the significant performance drops on mutated examples highlight a persistent reliance on memorization. The authors advocate for dynamic evaluation—synthesizing benchmarks on the fly—to ensure that AI progress is measured by genuine logical proficiency rather than statistical recall.

Summary updated

BibTeX
@inproceedings{xu2025re,
  author = {Xinnuo Xu and Rachel Lawrence and Kshitij Dubey and Atharva Pandey and Risa Ueno and Fabian Falck and Aditya V. Nori and Rahul Sharma and Amit Sharma and Javier González},
  title = {{RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=QJPl0DWajD},
}
Figure