Abstract
Large Language Models (LLMs) have revolutionized various domains but encounter substantial challenges in tackling optimization modeling tasks for Operations Research (OR), particularly when dealing with complex problems. In this work, we propose Step-Opt-Instruct, a framework that augments existing datasets and generates high-quality fine-tuning data tailored to optimization modeling. Step-Opt-Instruct employs iterative problem generation to systematically increase problem complexity and stepwise validation to rigorously verify data, preventing error propagation and ensuring the quality of the generated dataset. Leveraging this framework, we fine-tune open-source LLMs, including LLaMA-3-8B and Mistral-7B, to develop Step-Opt—a model that achieves state-of-the-art performance on benchmarks such as NL4OPT, MAMO, and IndustryOR. Extensive experiments demonstrate the superior performance of Step-Opt, especially in addressing complex OR tasks, with a notable 17.01% improvement in micro average accuracy on difficult problems.
Introduction
Operations Research (OR) is critical for data-driven decision-making in logistics, finance, and manufacturing. However, translating natural language problem descriptions into mathematical models and executable code remains a significant bottleneck. While LLMs show promise, they often struggle with the logical rigor required for OR, frequently hallucinating constraints or failing on multi-layered problems. Existing fine-tuning datasets are often limited in scale or quality, leading to a "garbage in, garbage out" cycle when using LLMs for data synthesis. Step-Opt addresses these challenges by introducing a structured, iterative approach to data generation and validation.
The Step-Opt-Instruct Framework
The core of our approach is Step-Opt-Instruct, a two-pillar framework designed to synthesize a high-fidelity dataset for OR modeling.
1. Iterative Problem Generation
Rather than generating complex problems in a single step, the framework evolves simple "seed" problems through two strategies:
- Complexity-Evolving: Incrementally adding constraints, adjusting parameters, or altering objectives to increase mathematical difficulty.
- Scope-Evolving: Transforming the problem domain (e.g., from healthcare to retail) or combining multiple distinct problems to broaden the model's contextual understanding.
2. Stepwise Validation Mechanism
To ensure data integrity, every generated sample undergoes a rigorous four-stage validation process:
- Description Checker: Verifies the completeness of the problem statement.
- Variables & Constraints Checkers: Ensures mathematical definitions align with the text.
- Program Checker: Actually executes the generated Python code (using
coptpy) to confirm it produces valid results. - Feedback Loops: Models are prompted to correct identified errors; if a sample fails after multiple attempts, it is discarded.
Experimental Evaluation
We fine-tuned LLaMA-3-8B and Mistral-7B using approximately 4,500 high-quality examples generated by Step-Opt-Instruct. Evaluation was conducted across four major benchmarks: NL4OPT, MAMO (EasyLP and ComplexLP), and IndustryOR. To ensure a fair comparison, the researchers manually corrected inaccuracies in the original benchmarks, fixing 78 errors in MAMO and 50 in IndustryOR.
Results
As shown in the performance comparison table, Step-Opt-LLaMA-3-8B achieves state-of-the-art results across all benchmarks. Notably, on the difficult MAMO ComplexLP task, Step-Opt reached 61.61% accuracy, significantly outperforming GPT-4 (40.28%) and previous specialized models like ORLM (38.39%).
Key Insights and Discussion
- Data Quality over Quantity: During synthesis, nearly 47% of generated samples were discarded by the validation checkers. This underscores the necessity of structured validation in technical domains where logical precision is paramount.
- Mathematical Scaffolding: Including formal mathematical models in the training data—not just code—acts as a structured "Chain-of-Thought," helping the model bridge the gap between natural language and executable scripts.
- Efficiency of Evolution: Modifying known-good problems through evolution proved more effective at maintaining logical consistency than generating complex problems from a blank page.
Conclusion
Step-Opt demonstrates that combining iterative problem refinement with rigorous stepwise validation can significantly enhance LLM performance in specialized technical domains like Operations Research. By outperforming much larger proprietary models, Step-Opt highlights the potential of high-quality, domain-specific data synthesis. Future work will focus on expanding the framework to include non-linear and stochastic programming, as well as improving solver-agnostic code generation.
Summary updated