Abstract
Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. Solving realistic optimization (OPT) problems in application scenarios requires advanced and applied mathematics ability. However, current OPT benchmarks that merely solve linear programming are far from complex realistic situations. In this work, we propose OptiBench, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs. OptiBench contains rich optimization problems, including linear and nonlinear programming with or without tabular data, which can comprehensively evaluate LLMs' solving ability. In our benchmark, LLMs are required to call a code solver to provide precise numerical answers. Furthermore, to alleviate the data scarcity for optimization problems, and to bridge the gap between open-source LLMs on a small scale (e.g., Llama-3-8b) and closed-source LLMs (e.g., GPT-4), we further propose a data synthesis method namely ReSocratic. Unlike general data synthesis methods that proceed from questions to answers, ReSocratic first incrementally synthesizes formatted optimization demonstration with mathematical formulations step by step and then back-translates the generated demonstrations into questions. Based on this, we synthesize the ReSocratic-29k dataset. We further conduct supervised fine-tuning with ReSocratic-29k on multiple open-source models. Experimental results show that ReSocratic-29k significantly improves the performance of open-source models.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in high school-level mathematics, their ability to navigate the complexities of real-world industrial optimization remains a significant challenge. Practical operations research (OR)—encompassing fields like supply chain management, logistics, and power grid scheduling—requires more than just basic logic; it demands the ability to translate messy, often tabular data into rigorous mathematical formulations.
Existing benchmarks for optimization modeling are frequently too narrow, focusing primarily on linear programming with clean, text-only inputs. To address these limitations, we introduce OptiBench, a rigorous new benchmark for end-to-end optimization tasks, and ReSocratic, a novel "reverse-engineering" data synthesis method designed to enhance the reasoning capabilities of open-source models.
OptiBench: A Comprehensive Optimization Benchmark
OptiBench is designed to evaluate LLMs across a spectrum of complexity. It consists of 605 high-quality questions categorized into four primary types: linear and nonlinear problems, both with and without tabular data. Unlike previous benchmarks, OptiBench requires models to generate executable Python code (utilizing the PySCIPOpt library) to produce precise numerical answers, rather than just providing textual mathematical formulations.
Our evaluation reveals that tabular data and nonlinearity represent the most significant hurdles for current models. LLMs often struggle to accurately map values from tables to mathematical variables, and the increased complexity of nonlinear solver syntax leads to higher failure rates in code generation.
ReSocratic: Enhancing Reasoning via Reverse Synthesis
High-quality optimization training data is scarce and expensive to produce. To bridge the performance gap between open-source models (like Llama-3-8B) and closed-source giants (like GPT-4), we developed ReSocratic.
Unlike traditional "forward" synthesis (Question → Answer), ReSocratic flips the process:
- Incremental Synthesis: The system starts with a seed idea and builds a logical mathematical "skeleton" step-by-step, defining variables, objectives, and constraints.
- Back-translation: Once the logic is verified, the LLM translates the mathematical formulation back into a natural language word problem.
- Verification: The generated code is automatically executed to ensure the problem is solvable and logically sound.
Using this method, we created ReSocratic-29k, a dataset of 29,000 synthetic samples for Supervised Fine-Tuning (SFT).
Experimental Results and Insights
Our experiments demonstrate that high-quality synthetic data can dramatically "up-skill" smaller models. After fine-tuning on ReSocratic-29k, Llama-3-8B-Instruct saw its accuracy on OptiBench jump from 13.6% to 51.1%. Even more strikingly, Llama-2-7B-Chat, which originally failed to solve a single problem (0% accuracy), improved to 30.6%.
While GPT-4o leads the benchmark with an overall zero-shot accuracy of 66.1%, it still faces challenges with complex nonlinear tabular problems. The primary bottleneck for all models remains the "Code Pass" rate; if a model cannot produce syntactically correct code for the solver, it cannot reach the correct numerical solution.
Conclusion
OptiBench and the ReSocratic method provide a new path forward for applying LLMs to complex optimization tasks. By focusing on end-to-end solving and rigorous reverse-engineered data synthesis, we can significantly narrow the gap between open-source and proprietary models. Future work will involve expanding ReSocratic to other reasoning domains and exploring more complex mixed-integer problems that reflect the most difficult real-world constraints.
Summary updated