Empirical Equilibria in Agent-based Economic systems with Learning agents

Kshama Dwarakanath, Svitlana Vyetrenko, Tucker BalcharXiv 2024 · DOI · arXiv

Introduction

Traditional macroeconomic models, such as Dynamic Stochastic General Equilibrium (DSGE), often rely on oversimplified assumptions like "representative agents" to maintain mathematical tractability. While Agent-Based Models (ABMs) offer a more flexible alternative for simulating complex, heterogeneous interactions, they are notoriously difficult to analyze for stability. This paper introduces a high-fidelity economic simulator that leverages the Policy Space Response Oracle (PSRO) algorithm to identify stable economic equilibria. By integrating reinforcement learning with empirical game theory, the researchers demonstrate that agents can reach a near-Nash equilibrium with significantly lower "regret" than standard independent multi-agent reinforcement learning (MARL) approaches.

Methodology

The authors developed a macroeconomic simulator based on ABIDES, an open-source discrete event simulator. The model includes four primary agent types:

  • Households: Optimize utility from consumption and savings while balancing labor.
  • Firms: Set prices and wages to maximize profit and minimize inventory risk.
  • Central Bank: Adjusts interest rates to hit inflation targets.
  • Government: Sets tax rates and redistributes wealth to maximize social welfare.

To ensure realism, the simulator is calibrated using real-world data from the US Bureau of Labor Statistics, the Federal Reserve Board, and the IRS. To find an equilibrium, the authors employ PSRO, an iterative process that maintains a "meta-strategy" pool of policies. In each epoch, reinforcement learning is used to find a "best response" for one agent type against the current meta-strategies of all others.

Training convergence of agent utilities across PSRO epochs, showing the iterative learning process.

Key Insights: Emergent Specialization and Stability

A central finding of this research is that stable equilibria facilitate the emergence of economic specialization. Under PSRO training, households with varying skill levels exhibited distinct labor patterns. For instance, a household more proficient at "Firm 1" dedicated significantly more labor hours to that firm compared to a less-skilled counterpart. This specialization is far more pronounced in PSRO-derived equilibria than in standard MARL, where the "moving target" problem of non-stationarity often blurs these distinctions.

Distribution of household labor hours and consumption, illustrating emergent specialization in stable states.

Furthermore, the PSRO-trained agents exhibited more "principled" behavior. The Central Bank, for example, learned to make decisive interest rate moves to control inflation, whereas independent MARL agents often adopted more sporadic and less effective strategies.

Results

The primary success metric used is "regret"—the potential gain an agent could achieve by unilaterally changing its strategy. Lower regret indicates a state closer to a Nash equilibrium. The key results include:

  • Regret Reduction: The PSRO-trained system achieved a total regret of 0.17 (0.01%), compared to 4.21 (0.22%) for independent MARL.
  • Inflation Control: PSRO strategies successfully achieved target inflation rates with significantly lower variance than MARL.
  • Tax Efficiency: The PSRO government learned to set lower tax rates while maintaining revenue, as households in a stable equilibrium were more willing to work.

Conclusion

While PSRO offers a robust framework for finding equilibria in complex ABMs, it faces challenges regarding computational complexity as the policy pool grows. Future work will focus on scaling this framework to thousands of heterogeneous agents and incorporating more complex economic components like international trade and supply chains.

Summary updated

Abstract

We present an agent-based simulator for economic systems with heterogeneous households, firms, central bank, and government agents. These agents interact to define production, consumption, and monetary flow. Each agent type has distinct objectives, such as households seeking utility from consumption and the central bank targeting inflation and production. We define this multi-agent economic system using an OpenAI Gym-style environment, enabling agents to optimize their objectives through reinforcement learning. Standard multi-agent reinforcement learning (MARL) schemes, like independent learning, enable agents to learn concurrently but do not address whether the resulting strategies are at equilibrium. This study integrates the Policy Space Response Oracle (PSRO) algorithm, which has shown superior performance over independent MARL in games with homogeneous agents, with economic agent-based modeling. We use PSRO to develop agent policies approximating Nash equilibria of the empirical economic game, thereby linking to economic equilibria. Our results demonstrate that PSRO strategies achieve lower regret values than independent MARL strategies in our economic system with four agent types. This work aims to bridge artificial intelligence, economics, and empirical game theory towards future research.
BibTeX
@misc{dwarakanath2024empirical,
  author = {Kshama Dwarakanath and Svitlana Vyetrenko and Tucker Balch},
  title = {{Empirical Equilibria in Agent-based Economic systems with Learning agents}},
  year = {2024},
  eprint = {2408.12038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.12038},
}
Figure