TL;DR
AutoOR is a scalable post-training pipeline that uses synthetic data generation and online reinforcement learning (RL) to train large language models to automatically formulate optimization problems from natural language. By leveraging solver execution feedback and a novel curriculum RL strategy, an 8B model post-trained with AutoOR matches or outperforms massive frontier models across linear, mixed-integer, and highly complex non-linear programming problems.
What Problem Does This Paper Address?
In industries like logistics, manufacturing, and energy, operational efficiency relies on solving complex mathematical optimization problems. However, translating a messy, real-world natural language description of a problem into solver-ready code (a process called autoformulation) requires highly specialized operations research (OR) expertise.
While large language models (LLMs) show promise in automating this translation, existing approaches suffer from major bottlenecks: public training data is scarce, synthetic datasets often have low accuracy, and distilling data from proprietary frontier models is expensive and hard to scale. Furthermore, highly non-linear optimization problems involving physical dynamics remain a massive blind spot—frontier models score near 0% on them, and standard reinforcement learning fails because models suffer from a "cold-start" problem, never receiving a single positive reward signal to kickstart learning.
Methods
AutoOR circumvents these challenges by turning autoformulation into a verifiable RL domain, exploiting a structural asymmetry: generating code from text is incredibly difficult, but verifying text against a structured mathematical template is cheap and scalable.
The pipeline consists of two main components:
- Scalable Synthetic Data Generation (Backtranslation): Instead of generating code from text, AutoOR instantiates standard optimization form templates (LP, MILP, NLP) with random valid parameters to generate correct-by-construction solver code and a structured "world descriptor." It then uses an LLM to backtranslate this descriptor into a natural language problem description, which is verified component-wise.
- RL Training with Verifiable Rewards: Using Group Relative Policy Optimization (GRPO), AutoOR trains a base model (Qwen3-8B) on the hard direction (description $\rightarrow$ code). Crucially, it bypasses the need for a learned reward model by executing the generated code directly in a solver (like Google OR-Tools or Gekko) and checking for runtime success, constraint feasibility, and objective optimality.
Overcoming the Cold Start: Curriculum RL
To tackle non-linear programming (NLP) problems governed by complex physics equations, the authors introduce a three-phase Curriculum RL strategy:
- Phase 1: The model is trained on easy problems with privileged information (the solver syntax and explicit constraint mappings are provided directly in the prompt) to bootstrap exploration.
- Phase 2: The privileged guidance is removed, forcing the model to rely on its internalized understanding of the solver syntax on the same easy problem distribution.
- Phase 3: The model is transitioned to the full, hard non-linear problem distribution.
Additionally, the authors introduce a Multi-Turn Training setup, teaching the model to recognize incomplete problem descriptions and ask targeted clarifying questions before generating code.
Datasets
The authors evaluated AutoOR across a robust suite of benchmarks:
- Established Benchmarks: NL4Opt, NL4LP, MAMO-Easy, MAMO-Complex, IndustryOR, and ComplexOR.
- Newly Introduced Benchmarks:
- Hard-LP and Hard-MIP: Designed with longer descriptions and more complex constraint structures.
- Pump-NLP: A highly challenging non-linear programming benchmark based on a real-world centrifugal pump network synthesis problem.
Key Insights and Ideas
- The Generation-Verification Gap: Optimization autoformulation is uniquely suited for Reinforcement Learning with Verifiable Rewards (RLVR). Because solvers provide deterministic, noise-free verification of mathematical correctness, models can learn entirely from environment feedback without human labels.
- Privileged Information as an RL Ladder: In environments where the initial success rate is absolute zero, standard RL gradients vanish. Providing temporary, "privileged" structural hints allows the model to find its first successful rollouts, which can then be progressively phased out.
- RL Generalizes Better Than SFT: While Supervised Fine-Tuning (SFT) often causes models to memorize specific training distributions and fail on out-of-distribution variants, online RL forces the model to explore the underlying mathematical structure, leading to superior generalization on longer, more complex descriptions.
Key Results
- Frontier-Level Performance from an 8B Model: Post-trained on purely synthetic data, AutoOR 8B consistently outperformed Gemini 2.5 Pro and achieved competitive or state-of-the-art results against Gemini 3 Pro across linear and mixed-integer benchmarks.
- Conquering Non-Linear Physics: On the brutal Pump-NLP benchmark, where Qwen3-8B, Gemini 2.5 Pro, and Gemini 3 Pro all scored 0%, AutoOR 8B achieved a remarkable 48.98% accuracy, unlocked entirely by the curriculum RL strategy.
- Conversational Error Recovery: In the multi-turn setting with missing information, an un-tuned model scored 0%. After two-turn GRPO training, AutoOR reached 66.75% accuracy by learning to ask highly specific questions (e.g., asking for a precise per-unit cost rather than a generic "give me more details").
Limitations and Future Work
While AutoOR marks a significant step forward, the authors note several limitations:
- Domain Specificity: The non-linear programming results are currently restricted to a single problem class (pump network synthesis).
- Coarse Reward Function: The reward system evaluates whether a formulation is correct, but it does not penalize formulation inefficiency (e.g., code that takes significantly longer for a solver to compute).
- API Dependency: The model generates code tailored to specific solver APIs (Google OR-Tools and Gekko) rather than solver-agnostic mathematical representations.
Future work will focus on expanding AutoOR to broader non-linear domains, optimizing for formulation efficiency, and scaling multi-turn training to support complex agentic tool use.
Summary updated