AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problems

Sumeet Ramesh Motwani, Chuan Du, Aleksander Petrov, Christopher Davis, Philip Torr, Antonio Papania-Davis, Weishi Yan

TL;DR

AutoOR is a scalable post-training pipeline that uses synthetic data generation and online reinforcement learning (RL) to train large language models to automatically formulate optimization problems from natural language. By leveraging solver execution feedback and a novel curriculum RL strategy, an 8B model post-trained with AutoOR matches or outperforms massive frontier models across linear, mixed-integer, and highly complex non-linear programming problems.

What Problem Does This Paper Address?

In industries like logistics, manufacturing, and energy, operational efficiency relies on solving complex mathematical optimization problems. However, translating a messy, real-world natural language description of a problem into solver-ready code (a process called autoformulation) requires highly specialized operations research (OR) expertise.

While large language models (LLMs) show promise in automating this translation, existing approaches suffer from major bottlenecks: public training data is scarce, synthetic datasets often have low accuracy, and distilling data from proprietary frontier models is expensive and hard to scale. Furthermore, highly non-linear optimization problems involving physical dynamics remain a massive blind spot—frontier models score near 0% on them, and standard reinforcement learning fails because models suffer from a "cold-start" problem, never receiving a single positive reward signal to kickstart learning.

Methods

AutoOR circumvents these challenges by turning autoformulation into a verifiable RL domain, exploiting a structural asymmetry: generating code from text is incredibly difficult, but verifying text against a structured mathematical template is cheap and scalable.

The pipeline consists of two main components:

  1. Scalable Synthetic Data Generation (Backtranslation): Instead of generating code from text, AutoOR instantiates standard optimization form templates (LP, MILP, NLP) with random valid parameters to generate correct-by-construction solver code and a structured "world descriptor." It then uses an LLM to backtranslate this descriptor into a natural language problem description, which is verified component-wise.
  2. RL Training with Verifiable Rewards: Using Group Relative Policy Optimization (GRPO), AutoOR trains a base model (Qwen3-8B) on the hard direction (description $\rightarrow$ code). Crucially, it bypasses the need for a learned reward model by executing the generated code directly in a solver (like Google OR-Tools or Gekko) and checking for runtime success, constraint feasibility, and objective optimality.
AutoOR Overview: Synthetic data generation and RL training pipeline.

Overcoming the Cold Start: Curriculum RL

To tackle non-linear programming (NLP) problems governed by complex physics equations, the authors introduce a three-phase Curriculum RL strategy:

  • Phase 1: The model is trained on easy problems with privileged information (the solver syntax and explicit constraint mappings are provided directly in the prompt) to bootstrap exploration.
  • Phase 2: The privileged guidance is removed, forcing the model to rely on its internalized understanding of the solver syntax on the same easy problem distribution.
  • Phase 3: The model is transitioned to the full, hard non-linear problem distribution.

Additionally, the authors introduce a Multi-Turn Training setup, teaching the model to recognize incomplete problem descriptions and ask targeted clarifying questions before generating code.

Datasets

The authors evaluated AutoOR across a robust suite of benchmarks:

  • Established Benchmarks: NL4Opt, NL4LP, MAMO-Easy, MAMO-Complex, IndustryOR, and ComplexOR.
  • Newly Introduced Benchmarks:
  • Hard-LP and Hard-MIP: Designed with longer descriptions and more complex constraint structures.
  • Pump-NLP: A highly challenging non-linear programming benchmark based on a real-world centrifugal pump network synthesis problem.
The centrifugal pump network synthesis problem used for the non-linear benchmark.

Key Insights and Ideas

  • The Generation-Verification Gap: Optimization autoformulation is uniquely suited for Reinforcement Learning with Verifiable Rewards (RLVR). Because solvers provide deterministic, noise-free verification of mathematical correctness, models can learn entirely from environment feedback without human labels.
  • Privileged Information as an RL Ladder: In environments where the initial success rate is absolute zero, standard RL gradients vanish. Providing temporary, "privileged" structural hints allows the model to find its first successful rollouts, which can then be progressively phased out.
  • RL Generalizes Better Than SFT: While Supervised Fine-Tuning (SFT) often causes models to memorize specific training distributions and fail on out-of-distribution variants, online RL forces the model to explore the underlying mathematical structure, leading to superior generalization on longer, more complex descriptions.

Key Results

  • Frontier-Level Performance from an 8B Model: Post-trained on purely synthetic data, AutoOR 8B consistently outperformed Gemini 2.5 Pro and achieved competitive or state-of-the-art results against Gemini 3 Pro across linear and mixed-integer benchmarks.
  • Conquering Non-Linear Physics: On the brutal Pump-NLP benchmark, where Qwen3-8B, Gemini 2.5 Pro, and Gemini 3 Pro all scored 0%, AutoOR 8B achieved a remarkable 48.98% accuracy, unlocked entirely by the curriculum RL strategy.
Reward curves showing how the curriculum RL strategy successfully navigates the three training phases.
  • Conversational Error Recovery: In the multi-turn setting with missing information, an un-tuned model scored 0%. After two-turn GRPO training, AutoOR reached 66.75% accuracy by learning to ask highly specific questions (e.g., asking for a precise per-unit cost rather than a generic "give me more details").

Limitations and Future Work

While AutoOR marks a significant step forward, the authors note several limitations:

  • Domain Specificity: The non-linear programming results are currently restricted to a single problem class (pump network synthesis).
  • Coarse Reward Function: The reward system evaluates whether a formulation is correct, but it does not penalize formulation inefficiency (e.g., code that takes significantly longer for a solver to compute).
  • API Dependency: The model generates code tailored to specific solver APIs (Google OR-Tools and Gekko) rather than solver-agnostic mathematical representations.

Future work will focus on expanding AutoOR to broader non-linear domains, optimizing for formulation efficiency, and scaling multi-turn training to support complex agentic tool use.

Summary updated

Abstract

Optimization problems are central to decision-making in manufacturing, logistics, scheduling, and other industrial settings. Translating complicated descriptions of these problems into solver-ready formulations requires specialized operations research (OR) expertise, making it hard to scale. We present AutoOR, a scalable synthetic data generation and reinforcement learning pipeline that trains LLMs to autoformulate optimization problems specified in natural language across linear, mixed-integer, and non-linear categories. AutoOR generates verified training data from standard optimization forms and uses solver execution feedback as the reward signal for RL post-training. AutoOR applied to an 8B model achieves state-of-the-art or competitive results across six established OR benchmarks, matching significantly larger frontier models. For a non-linear problem class involving physical dynamics, where frontier models score near 0%, we introduce a curriculum RL strategy that bootstraps from limited initial training data to make this class tractable for post-training. We believe that methods such as AutoOR can significantly accelerate industrial decision-making with AI.
BibTeX
@inproceedings{motwani2026autoor,
  author = {Sumeet Ramesh Motwani and Chuan Du and Aleksander Petrov and Christopher Davis and Philip Torr and Antonio Papania-Davis and Weishi Yan},
  title = {{AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problems}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=BRISZRO6Qh},
}
Figure