TL;DR
OR-License is a new, high-stakes benchmark designed to evaluate whether LLM agents can function as real-world Operations Research (OR) practitioners. Unlike previous benchmarks that use "clean" textbook problems, OR-License uses de-identified materials from $3.68 million worth of actual industrial projects and introduces ORSim, a client simulator that forces agents to elicit missing business requirements. The results show a significant "feasibility gap": while frontier models can sometimes out-optimize humans on specific tasks, they frequently fail to produce solutions that satisfy basic real-world constraints.
What Problem Does This Paper Address?
In the real world, Operations Research isn't just about solving a math equation; it’s about finding the equation. A practitioner must take messy, incomplete business briefs, talk to stakeholders to uncover hidden constraints, and then implement a solver that actually works.
Existing benchmarks (like NL4Opt or IndustryOR) typically provide agents with self-contained, perfectly described problems. This ignores the most difficult part of the job: requirement elicitation and problem formulation. As shown in Figure 1, there are three major gaps in current evaluations: a lack of practical relevance, the omission of information acquisition, and a lack of grounding in actual client-accepted deliveries.
Methods
The authors developed an end-to-end evaluation framework that treats the LLM as a "hired consultant."
- Agentic Workflow: Agents are given an initial brief and data files. They operate in a Bash-based environment (Claude Code or Codex) where they can write code, run solvers (like Gurobi or HiGHS), and inspect outputs.
- ORSim (The Client Simulator): To mimic real-world ambiguity, the authors created ORSim. If an agent realizes information is missing (e.g., "Is overtime allowed?"), it must use a specific
asktool. ORSim uses a "flat selector" architecture to provide factual business answers from a hidden catalog without offering any mathematical or modeling advice. - Deployment-Adjusted Quality (DAQ): The paper introduces a "feasibility-gated" metric. In OR, a solution that is 99% optimal but violates a hard constraint (like overloading a truck) is worth zero. DAQ assigns a score of 0 to any infeasible delivery and scales feasible ones against the original expert solution.
Datasets
The OR-License dataset consists of 31 instances across 10 industrial scenarios. These aren't toy problems; they are derived from paid engagements in sectors like food manufacturing, electronics, and financial investments. The total project value of the source material is $3.68M, representing months of human expert effort.
Key Insights and Ideas
- The Feasibility Bottleneck: The primary reason LLMs fail isn't poor optimization—it's failing to meet the constraints. When models do find a feasible solution, their objective quality often matches or even slightly exceeds human experts. However, their "Strict Feasibility" rates vary wildly (from 45% to 88%).
- Information "Front-Loading": Agents tend to ask all their questions at the very beginning (Stage S1). They rarely revisit their assumptions or ask for clarification later in the process, even when their code fails or produces weird results.
- Scientific Inquiry Gaps: The authors tracked "HCAEU" chains (Hypothesis, Explanation, Action, Evidence, Update). They found that while models are decent at interpreting evidence, they often fail to close the loop by using that evidence to actually update their solution.
- Expert vs. Agent Strategy: Human experts use more sophisticated "problem-structure exploitation," such as autonomous decomposition and valid tightening of the search space. Agents, conversely, rely more heavily on "greedy" rules or simple solver invocations.
Key Results
The study evaluated 11 frontier models. Grok 4.6 emerged as the strongest performer in terms of feasibility (88.39%) and DAQ (0.87), though it still fell short of the human expert reference (DAQ = 1.0).
As shown in the table above, models like GPT-5.6 Sol and Claude Opus 5 achieved over 100% "Conditional Quality" (meaning they found better mathematical solutions than experts on specific tasks), but their overall DAQ was dragged down by instances where they failed to produce a feasible result at all.
Limitations and Future Work
- Manual Annotation: The trajectory analysis (understanding why a model failed) requires significant manual effort from OR experts, making it hard to scale.
- Simulator Boundaries: While ORSim is robust, it still operates within a "closed catalog" of facts. Real-world elicitation is often more open-ended.
- Long-Horizon Reliability: The findings suggest that current agents lack a "feedback-responsive process." Future work should focus on agents that can link experimental evidence to self-improvement throughout a long-horizon project lifecycle.
Summary updated