OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen, Gaoyang Dai, Xing He, Aiqian Li, Fengchi Yuan, Aochuan Chen, Chenyi Zi, Yuhan Li, Hongchuan Zeng, Yimo Yan, Nuo Chen, Jie Fu, Zuo-Jun Max Shen, Yong-Hong Kuo, Jia LiarXiv 2026 · arXiv

TL;DR

OR-License is a new, high-stakes benchmark designed to evaluate whether LLM agents can function as real-world Operations Research (OR) practitioners. Unlike previous benchmarks that use "clean" textbook problems, OR-License uses de-identified materials from $3.68 million worth of actual industrial projects and introduces ORSim, a client simulator that forces agents to elicit missing business requirements. The results show a significant "feasibility gap": while frontier models can sometimes out-optimize humans on specific tasks, they frequently fail to produce solutions that satisfy basic real-world constraints.

What Problem Does This Paper Address?

In the real world, Operations Research isn't just about solving a math equation; it’s about finding the equation. A practitioner must take messy, incomplete business briefs, talk to stakeholders to uncover hidden constraints, and then implement a solver that actually works.

Existing benchmarks (like NL4Opt or IndustryOR) typically provide agents with self-contained, perfectly described problems. This ignores the most difficult part of the job: requirement elicitation and problem formulation. As shown in Figure 1, there are three major gaps in current evaluations: a lack of practical relevance, the omission of information acquisition, and a lack of grounding in actual client-accepted deliveries.

Figure 1: Three gaps and corresponding improvements in OR-LICENSE. Each column links an evaluation gap (top) to its improvement (bottom). The client dialogue is illustrative.

Methods

The authors developed an end-to-end evaluation framework that treats the LLM as a "hired consultant."

  1. Agentic Workflow: Agents are given an initial brief and data files. They operate in a Bash-based environment (Claude Code or Codex) where they can write code, run solvers (like Gurobi or HiGHS), and inspect outputs.
  2. ORSim (The Client Simulator): To mimic real-world ambiguity, the authors created ORSim. If an agent realizes information is missing (e.g., "Is overtime allowed?"), it must use a specific ask tool. ORSim uses a "flat selector" architecture to provide factual business answers from a hidden catalog without offering any mathematical or modeling advice.
  3. Deployment-Adjusted Quality (DAQ): The paper introduces a "feasibility-gated" metric. In OR, a solution that is 99% optimal but violates a hard constraint (like overloading a truck) is worth zero. DAQ assigns a score of 0 to any infeasible delivery and scales feasible ones against the original expert solution.
Figure 2: The OR-LICENSE evaluation framework. Experts curate tasks and reference solutions from real-world OR projects. Agents may query ORSIM throughout problem solving. Final submis- sions undergo hidden evaluation of feasibility and expert-relative solution quality.

Datasets

The OR-License dataset consists of 31 instances across 10 industrial scenarios. These aren't toy problems; they are derived from paid engagements in sectors like food manufacturing, electronics, and financial investments. The total project value of the source material is $3.68M, representing months of human expert effort.

Figure 3: Overview of the dataset. (a): The two-ring donut summarizes the covered industries (outer ring) and OR problem types (inner ring). Industry categories are based on the three-digit subsectors of the 2022 North American Industry Classification System (NAICS) (Office of Management and Budget, 2022), with selected labels shortened for readability. (b): Source-engagement statistics summarize the financial value and human effort associated with the original client projects. (c): The benchmark covers six core OR expert capabilities reflected in the curated tasks.

Key Insights and Ideas

  • The Feasibility Bottleneck: The primary reason LLMs fail isn't poor optimization—it's failing to meet the constraints. When models do find a feasible solution, their objective quality often matches or even slightly exceeds human experts. However, their "Strict Feasibility" rates vary wildly (from 45% to 88%).
  • Information "Front-Loading": Agents tend to ask all their questions at the very beginning (Stage S1). They rarely revisit their assumptions or ask for clarification later in the process, even when their code fails or produces weird results.
  • Scientific Inquiry Gaps: The authors tracked "HCAEU" chains (Hypothesis, Explanation, Action, Evidence, Update). They found that while models are decent at interpreting evidence, they often fail to close the loop by using that evidence to actually update their solution.
  • Expert vs. Agent Strategy: Human experts use more sophisticated "problem-structure exploitation," such as autonomous decomposition and valid tightening of the search space. Agents, conversely, rely more heavily on "greedy" rules or simple solver invocations.

Key Results

The study evaluated 11 frontier models. Grok 4.6 emerged as the strongest performer in terms of feasibility (88.39%) and DAQ (0.87), though it still fell short of the human expert reference (DAQ = 1.0).

Table 1: Main results under Claude Code on 31 tasks with five runs per model–task pair (%). Means weight tasks equally. For feasibility and DAQ, ± reports the mean within-task sample standard deviation; quality and its spread are DAQ and its spread divided by mean feasibility. Right: task- equal reliability–quality profiles; the expert reference is 1.

As shown in the table above, models like GPT-5.6 Sol and Claude Opus 5 achieved over 100% "Conditional Quality" (meaning they found better mathematical solutions than experts on specific tasks), but their overall DAQ was dragged down by instances where they failed to produce a feasible result at all.

Limitations and Future Work

  • Manual Annotation: The trajectory analysis (understanding why a model failed) requires significant manual effort from OR experts, making it hard to scale.
  • Simulator Boundaries: While ORSim is robust, it still operates within a "closed catalog" of facts. Real-world elicitation is often more open-ended.
  • Long-Horizon Reliability: The findings suggest that current agents lack a "feedback-responsive process." Future work should focus on agents that can link experimental evidence to self-improvement throughout a long-horizon project lifecycle.

Summary updated

Abstract

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.
BibTeX
@misc{wang2026or,
  author = {Peisong Wang and Feixue Liu and Yize Meng and Bowen Liu and Ruoyu Chen and Gaoyang Dai and Xing He and Aiqian Li and Fengchi Yuan and Aochuan Chen and Chenyi Zi and Yuhan Li and Hongchuan Zeng and Yimo Yan and Nuo Chen and Jie Fu and Zuo-Jun Max Shen and Yong-Hong KUO and Jia Li},
  title = {{OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners}},
  year = {2026},
  eprint = {2610.24087},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.24087},
}
Figure