TL;DR
OutfitterBench is a new benchmark that evaluates LLM agents on their ability to manage a simulated outdoor rental business over a 200-day horizon. The study finds that while frontier models can perform basic tasks, most struggle with "commitment coordination"—the ability to keep future plans feasible while weighing the economic value of competing resource uses—often falling behind simple rule-based baselines.
What Problem Does This Paper Address?
As LLM agents move from simple chat interfaces to autonomous operators, they face a fundamental challenge: long-horizon resource allocation. In the real world, a decision made today (like accepting a rental booking for next week) creates a future constraint. An agent must not only ensure that the equipment is available and cleaned in time for that booking but also decide if that booking is more valuable than potential "walk-in" customers who might show up in the meantime.
Existing benchmarks often focus on short-term tool use or static planning. This paper addresses the gap in evaluating how agents maintain intertemporal feasibility (keeping successive decisions compatible) and intertemporal valuation (reassessing the value of resources as opportunities change) over hundreds of simulated steps.
Methods
The authors designed OutfitterBench, a simulation where an agent acts as a rental manager in Beijing.
- The Task: Manage a shop for 200 days, handling ~1,100 reservation requests and uncertain walk-in demand.
- The Constraints: A fixed capital ceiling for purchasing equipment, variable cleaning/repair times, and weather-dependent demand.
- Agent Capabilities: Agents use 43 specialized tools to inspect assets, set prices, manage occupancy caps, and negotiate with customers.
- Evaluation: Performance is measured by "Terminal Business Value"—the sum of cash and equipment resale value minus liabilities at the end of the season.
To look under the hood, the authors developed "process diagnostics" to track why agents fail. These include tracking "Pending-booking gaps" (instances where an agent lacks the gear to fulfill a promise) and "Action-induced new gaps" (when a new decision, like a walk-in rental, breaks a previous commitment).
Datasets
- OutfitterBench: The primary contribution, featuring a 46-product catalog derived from real Taobao listings and historical Beijing weather data to drive realistic demand patterns.
- Models Evaluated: Seven frontier LLM configurations, including GPT 6.1 Sol, Claude Opus 5.5, and DeepSeek V4.1 Flash.
- Baseline: A rule-based "Calendar" policy that follows a fixed replenishment strategy and a simple availability check with a one-day cleaning buffer.
Key Insights and Ideas
- The "Calendar" Hurdle: A simple, consistent rule-based policy (Calendar) outperformed five out of the seven LLM agents. This suggests that for long-horizon tasks, consistency and basic logic are often more valuable than the sophisticated but erratic reasoning of current LLMs.
- Inconsistent Logic: Trajectory analysis revealed that agents often "knew" the rules (e.g., equipment needs one day for cleaning) but failed to apply them when schedules became crowded. Some agents would double-book assets even when the conflict was clearly visible in their records.
- Volume $\neq$ Value: Some agents (like DeepSeek and Opus) served significantly more bookings than the top-performing agent (Sol) but ended up with lower business value. They often charged too little or incurred high cancellation penalties, failing to capture the true economic value of their limited inventory.
- The Cost of Protection: Agents often struggle with "capacity protection." For example, an agent might block a tent for a future booking so early that they miss out on several profitable short-term rentals that could have been completed and cleaned before the reservation arrived.
Key Results
- Top Performer: Only GPT 6.1 Sol consistently outperformed the rule-based baseline, achieving a mean value creation of +2,556.78 CNY compared to the baseline's +1,119.51 CNY.
- Reliability Gaps: Sol maintained a schedule reliability of ~96%, whereas lower-performing agents like GLM 5.2 dropped to ~73%, leading to massive compensation costs (over 1,500 CNY in some cases).
- Capital Productivity: Sol generated significantly higher returns per unit of capital (1.38 CNY per 100 capital-days) than its closest competitors (~0.97 CNY), demonstrating superior resource valuation.
Limitations and Future Work
The authors acknowledge several limitations:
- Rule-Based Customers: Currently, customer behavior follows fixed probabilistic models rather than being LLM-driven, which might miss some nuances of human negotiation.
- Sample Size: The evaluation used only two world seeds per agent due to the high cost of long-horizon runs.
- Tooling: Agents were restricted to a fixed set of business tools. Future work could explore "coding agents" that can write their own scripts to manage complex schedules.
OutfitterBench serves as a sobering reminder that while LLMs are excellent at following immediate instructions, "playing the long game" in resource management remains a significant frontier for AI research.
Summary updated