EconEvals: Benchmarks and Litmus Tests for LLM Agents in Unknown Environments

Sara Fish, Julia Shephard, Minkai Li, Ran I. Shorrer, Yannai A. Gonczarowski

Abstract

We develop benchmarks for LLM agents that act in, learn from, and strategize in unknown environments, the specifications of which the LLM agent must learn over time from deliberate exploration. Our benchmarks consist of decision-making tasks derived from key problems in economics. To forestall saturation, the benchmark tasks are synthetically generated with scalable difficulty levels. Additionally, we propose litmus tests, a new kind of quantitative measure for LLMs and LLM agents. Unlike benchmarks, litmus tests quantify differences in character, values, and tendencies of LLMs and LLM agents, by considering their behavior when faced with tradeoffs (e.g., efficiency versus equality) where there is no objectively right or wrong behavior. Overall, our benchmarks and litmus tests assess the abilities and tendencies of LLM agents in tackling complex economic problems in diverse settings spanning procurement, scheduling, task allocation, and pricing -- applications that should grow in importance as such agents are further integrated into the economy.

Introduction

Standard Large Language Model (LLM) benchmarks, such as MMLU or GSM8K, are typically static, one-shot evaluations. While useful for assessing general knowledge and reasoning, they fail to capture the complexities of real-world economic agents who must operate in dynamic environments. These agents must learn from feedback over time and navigate "open-ended tradeoffs" where a single "correct" answer may not exist. For example, a manager assigning tasks must often choose between maximizing profit (efficiency) and ensuring equal pay (equality).

To address this gap, we introduce EconEvals, a framework designed to measure both learning-in-context (capabilities) and subjective decision-making tendencies (character).

The EconEvals Framework

The framework utilizes two distinct evaluation styles to provide a holistic view of an LLM agent's performance and behavior:

  1. Benchmarks (Capabilities): These assess a model's ability to optimize a well-defined goal, such as maximizing profit, over 100 "periods." The environment's rules—such as demand curves or worker preferences—are initially unknown, requiring the agent to use tools to gather data and explore the space.
  2. Litmus Tests (Tendencies): These quantify how a model navigates conflicting objectives. To ensure meaningful results, we report a triplet of scores: a Litmus Score (the model's position on a tradeoff spectrum), a Reliability Score (consistency across runs), and a Competency Score (a sanity check to ensure the model can optimize a single goal when instructed).
The interaction loop between the LLM agent and the economic environment.

Experimental Environments

We developed several synthetic environments to test these agents, focusing on core economic problems. Each environment provides the agent with specific "action" tools to influence the state and "getter" tools to retrieve information.

  • Procurement: A bin-packing task where agents must manage a budget to acquire items of unknown quality.
  • Scheduling: Producing a stable matching between workers and tasks with unknown preferences.
  • Pricing: Setting profit-maximizing prices in a non-stationary (changing) market.
Summary of action and getter tools available to agents in each environment.

Additionally, we designed specific Litmus Scenarios to test tradeoffs: "Patience vs. Impatience" (intertemporal choice), "Efficiency vs. Equality" (task allocation), and "Collusiveness vs. Competitiveness" (multi-agent pricing).

Key Insights and Results

Rapid Evolution of Capabilities

Economic decision-making capabilities are improving at a remarkable pace. Across the models tested, hard procurement scores increased by an average of 41 percentage points per year. Reasoning-focused models, such as OpenAI’s o-series and Gemini 3, significantly outperformed others in calculation-heavy tasks. These models demonstrated more sophisticated strategies in their internal "notes," often solving linear equations rather than relying on simple trial-and-error.

The Emergence of Model "Character"

Our litmus tests revealed that models exhibit distinct and stable "personalities." For instance, Gemini 3 Pro Preview emerged as highly competent but also notably "impatient" and "efficiency-focused" compared to its peers. Interestingly, a model’s preference for "Equality" in our tests correlated strongly with external measures of U.S. Democratic party political bias, suggesting that economic choices can reflect broader underlying values.

Robustness of Tendencies

Unlike many LLM behaviors that are sensitive to minor prompt changes, these economic tendencies proved remarkably stable. In the patience test, the specific model choice explained 97.9% of the variance in results, while superficial changes to the prompt (like the user's name) had negligible impact.

Conclusion

As LLM agents are increasingly integrated into supply chains, financial systems, and organizational management, understanding their inherent biases and capabilities becomes critical. EconEvals provides a necessary toolset for auditing these agents. While current evaluations are computationally expensive due to their multi-turn nature, they offer a vital look at how these models will behave when faced with the complex, value-laden tradeoffs of the real economy.

Summary updated

Abstract

We develop benchmarks for LLM agents that act in, learn from, and strategize in unknown environments, the specifications of which the LLM agent must learn over time from deliberate exploration. Our benchmarks consist of decision-making tasks derived from key problems in economics. To forestall saturation, the benchmark tasks are synthetically generated with scalable difficulty levels. Additionally, we propose litmus tests, a new kind of quantitative measure for LLMs and LLM agents. Unlike benchmarks, litmus tests quantify differences in character, values, and tendencies of LLMs and LLM agents, by considering their behavior when faced with tradeoffs (e.g., efficiency versus equality) where there is no objectively right or wrong behavior. Overall, our benchmarks and litmus tests assess the abilities and tendencies of LLM agents in tackling complex economic problems in diverse settings spanning procurement, scheduling, task allocation, and pricing -- applications that should grow in importance as such agents are further integrated into the economy.
BibTeX
@inproceedings{fish2025econevals,
  author = {Sara Fish and Julia Shephard and Minkai Li and Ran I. Shorrer and Yannai A. Gonczarowski},
  title = {{EconEvals: Benchmarks and Litmus Tests for LLM Agents in Unknown Environments}},
  booktitle = {MoFA @ ICML},
  year = {2025},
  url = {https://openreview.net/forum?id=OZhOYPva24},
}

Loading related content...

Figure