OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling

Zhicheng Yang, Yinya Huang, Wei Shi, Liang Feng, Linqi Song, Yiwei Wang, Xiaodan Liang, Jing Tang, Liang, Xiaodan, Tang, Jing

Abstract

Large language models (LLMs) have demonstrated significant potential in mathematical reasoning, yet solving real-world optimization (OPT) problems remains a formidable challenge. These problems require translating natural language into precise mathematical models and subsequently generating executable code. Current benchmarks often oversimplify these tasks, focusing primarily on linear programming. This paper introduces OptiBench, a comprehensive benchmark for end-to-end optimization problem-solving featuring linear and nonlinear programming with tabular data. To address the scarcity of high-quality training data, we propose ReSocratic, a "reverse" data synthesis method that generates mathematical formulations before back-translating them into natural language questions. Our results show that fine-tuning open-source models on the resulting ReSocratic-29k dataset significantly bridges the performance gap with closed-source models like GPT-4.

The Challenge of Optimization Modeling

Optimization problems are ubiquitous in industry, from supply chain logistics to power grid scheduling. For an LLM, solving these problems is a complex two-stage hurdle:

  1. Modeling: Interpreting a messy natural language description—often involving complex tabular data like price lists or distance matrices—and formulating it into precise mathematical equations.
  2. Coding: Writing a script to interface with specialized solvers (e.g., PySCIPOpt) to find numerical answers.

Existing benchmarks typically focus on simple linear programming, neglecting the tabular data and nonlinear constraints that define real-world industry problems. This creates a significant evaluation gap and a data bottleneck for training smaller, open-source models, which often lack the specialized reasoning required for these tasks.

OptiBench: A Comprehensive Benchmark

To provide a more rigorous evaluation, we developed OptiBench, consisting of 605 manually verified questions. OptiBench categorizes problems into four quadrants based on linearity and the presence of tabular data, providing a comprehensive map of an LLM's mathematical reasoning capabilities.

OptiBench covers four distinct categories of optimization problems.

Unlike previous benchmarks, OptiBench requires "End-to-End" solving. This means the model must provide the final numerical answer, ensuring that every link in the reasoning chain—from textual interpretation to code execution—is correct.

ReSocratic: Reverse Socratic Synthesis

High-quality optimization data is incredibly scarce and expensive to annotate. Traditional "forward" synthesis (Question $\rightarrow$ Answer) often leads to hallucinations or logical inconsistencies when LLMs attempt to write complex math problems from scratch. We propose ReSocratic, which flips this process on its head:

  • Seed Formulation: The process starts with a mathematical demonstration (the "answer").
  • Incremental Synthesis: An LLM builds a new, formatted mathematical model step-by-step.
  • Back-Translation: The math model is converted into a natural language story (the question).
  • Code Generation: The corresponding Python solver code is generated to match the story.

This "reverse" approach ensures that the textual question is perfectly aligned with the underlying mathematical logic, avoiding the "logical drift" common in other synthesis methods. Using this method, we created the ReSocratic-29k dataset.

Bridging the Performance Gap

Our experiments reveal that tabular data and nonlinear constraints are the "final bosses" for current LLMs. Even state-of-the-art models like GPT-4o struggle, achieving an overall accuracy of 66.1% on OptiBench.

Examples of optimization targets and constraints found in OptiBench.

However, the impact of the ReSocratic-29k dataset on open-source models is profound. Llama-3-8B-Instruct, which initially scored a dismal 13.6% in few-shot settings, improved to 51.1% after supervised fine-tuning (SFT) on our synthetic data. Notably, in the "Linear with Table" category, the fine-tuned Llama-3-8B (63.5%) actually outperformed GPT-4 (62.5%), proving that high-quality synthetic data can effectively compensate for smaller model parameters.

Conclusion

OptiBench and ReSocratic provide the tools necessary to measure and improve LLM performance in the critical domain of optimization. While models still struggle with specific solver syntax and formulaic storytelling, these innovations empower open-source models to tackle sophisticated mathematical reasoning tasks that were previously the sole domain of massive closed-source systems.

Summary updated

Abstract

Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. Solving realistic optimization (OPT) problems in application scenarios requires advanced and applied mathematics ability. However, current OPT benchmarks that merely solve linear programming are far from complex realistic situations. In this work, we propose OptiBench, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs. OptiBench contains rich optimization problems, including linear and nonlinear programming with or without tabular data, which can comprehensively evaluate LLMs' solving ability. In our benchmark, LLMs are required to call a code solver to provide precise numerical answers. Furthermore, to alleviate the data scarcity for optimization problems, and to bridge the gap between open-source LLMs on a small scale (e.g., Llama-3-8b) and closed-source LLMs (e.g., GPT-4), we further propose a data synthesis method namely ReSocratic. Unlike general data synthesis methods that proceed from questions to answers, \ReSocratic first incrementally synthesizes formatted optimization demonstration with mathematical formulations step by step and then back-translates the generated demonstrations into questions. Based on this, we synthesize the ReSocratic-29k dataset. We further conduct supervised fine-tuning with ReSocratic-29k on multiple open-source models. Experimental results show that ReSocratic-29k significantly improves the performance of open-source models.
BibTeX
@inproceedings{yang2025optibench,
  author = {Zhicheng Yang and Yinya Huang and Wei Shi and Liang Feng and Linqi Song and Yiwei Wang and Xiaodan Liang and Jing Tang and Liang, Xiaodan and Tang, Jing},
  title = {{OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=fsDZwS49uY},
}
Figure