Abstract
Large language models (LLMs) have demonstrated significant potential in mathematical reasoning, yet solving real-world optimization (OPT) problems remains a formidable challenge. These problems require translating natural language into precise mathematical models and subsequently generating executable code. Current benchmarks often oversimplify these tasks, focusing primarily on linear programming. This paper introduces OptiBench, a comprehensive benchmark for end-to-end optimization problem-solving featuring linear and nonlinear programming with tabular data. To address the scarcity of high-quality training data, we propose ReSocratic, a "reverse" data synthesis method that generates mathematical formulations before back-translating them into natural language questions. Our results show that fine-tuning open-source models on the resulting ReSocratic-29k dataset significantly bridges the performance gap with closed-source models like GPT-4.
The Challenge of Optimization Modeling
Optimization problems are ubiquitous in industry, from supply chain logistics to power grid scheduling. For an LLM, solving these problems is a complex two-stage hurdle:
- Modeling: Interpreting a messy natural language description—often involving complex tabular data like price lists or distance matrices—and formulating it into precise mathematical equations.
- Coding: Writing a script to interface with specialized solvers (e.g., PySCIPOpt) to find numerical answers.
Existing benchmarks typically focus on simple linear programming, neglecting the tabular data and nonlinear constraints that define real-world industry problems. This creates a significant evaluation gap and a data bottleneck for training smaller, open-source models, which often lack the specialized reasoning required for these tasks.
OptiBench: A Comprehensive Benchmark
To provide a more rigorous evaluation, we developed OptiBench, consisting of 605 manually verified questions. OptiBench categorizes problems into four quadrants based on linearity and the presence of tabular data, providing a comprehensive map of an LLM's mathematical reasoning capabilities.
Unlike previous benchmarks, OptiBench requires "End-to-End" solving. This means the model must provide the final numerical answer, ensuring that every link in the reasoning chain—from textual interpretation to code execution—is correct.
ReSocratic: Reverse Socratic Synthesis
High-quality optimization data is incredibly scarce and expensive to annotate. Traditional "forward" synthesis (Question $\rightarrow$ Answer) often leads to hallucinations or logical inconsistencies when LLMs attempt to write complex math problems from scratch. We propose ReSocratic, which flips this process on its head:
- Seed Formulation: The process starts with a mathematical demonstration (the "answer").
- Incremental Synthesis: An LLM builds a new, formatted mathematical model step-by-step.
- Back-Translation: The math model is converted into a natural language story (the question).
- Code Generation: The corresponding Python solver code is generated to match the story.
This "reverse" approach ensures that the textual question is perfectly aligned with the underlying mathematical logic, avoiding the "logical drift" common in other synthesis methods. Using this method, we created the ReSocratic-29k dataset.
Bridging the Performance Gap
Our experiments reveal that tabular data and nonlinear constraints are the "final bosses" for current LLMs. Even state-of-the-art models like GPT-4o struggle, achieving an overall accuracy of 66.1% on OptiBench.
However, the impact of the ReSocratic-29k dataset on open-source models is profound. Llama-3-8B-Instruct, which initially scored a dismal 13.6% in few-shot settings, improved to 51.1% after supervised fine-tuning (SFT) on our synthetic data. Notably, in the "Linear with Table" category, the fine-tuned Llama-3-8B (63.5%) actually outperformed GPT-4 (62.5%), proving that high-quality synthetic data can effectively compensate for smaller model parameters.
Conclusion
OptiBench and ReSocratic provide the tools necessary to measure and improve LLM performance in the critical domain of optimization. While models still struggle with specific solver syntax and formulaic storytelling, these innovations empower open-source models to tackle sophisticated mathematical reasoning tasks that were previously the sole domain of massive closed-source systems.
Summary updated