Introduction
Traditional macroeconomic models, such as Dynamic Stochastic General Equilibrium (DSGE), often rely on oversimplified assumptions like "representative agents" to maintain mathematical tractability. While Agent-Based Models (ABMs) offer a more flexible alternative for simulating complex, heterogeneous interactions, they are notoriously difficult to analyze for stability. This paper introduces a high-fidelity economic simulator that leverages the Policy Space Response Oracle (PSRO) algorithm to identify stable economic equilibria. By integrating reinforcement learning with empirical game theory, the researchers demonstrate that agents can reach a near-Nash equilibrium with significantly lower "regret" than standard independent multi-agent reinforcement learning (MARL) approaches.
Methodology
The authors developed a macroeconomic simulator based on ABIDES, an open-source discrete event simulator. The model includes four primary agent types:
- Households: Optimize utility from consumption and savings while balancing labor.
- Firms: Set prices and wages to maximize profit and minimize inventory risk.
- Central Bank: Adjusts interest rates to hit inflation targets.
- Government: Sets tax rates and redistributes wealth to maximize social welfare.
To ensure realism, the simulator is calibrated using real-world data from the US Bureau of Labor Statistics, the Federal Reserve Board, and the IRS. To find an equilibrium, the authors employ PSRO, an iterative process that maintains a "meta-strategy" pool of policies. In each epoch, reinforcement learning is used to find a "best response" for one agent type against the current meta-strategies of all others.
Key Insights: Emergent Specialization and Stability
A central finding of this research is that stable equilibria facilitate the emergence of economic specialization. Under PSRO training, households with varying skill levels exhibited distinct labor patterns. For instance, a household more proficient at "Firm 1" dedicated significantly more labor hours to that firm compared to a less-skilled counterpart. This specialization is far more pronounced in PSRO-derived equilibria than in standard MARL, where the "moving target" problem of non-stationarity often blurs these distinctions.
Furthermore, the PSRO-trained agents exhibited more "principled" behavior. The Central Bank, for example, learned to make decisive interest rate moves to control inflation, whereas independent MARL agents often adopted more sporadic and less effective strategies.
Results
The primary success metric used is "regret"—the potential gain an agent could achieve by unilaterally changing its strategy. Lower regret indicates a state closer to a Nash equilibrium. The key results include:
- Regret Reduction: The PSRO-trained system achieved a total regret of 0.17 (0.01%), compared to 4.21 (0.22%) for independent MARL.
- Inflation Control: PSRO strategies successfully achieved target inflation rates with significantly lower variance than MARL.
- Tax Efficiency: The PSRO government learned to set lower tax rates while maintaining revenue, as households in a stable equilibrium were more willing to work.
Conclusion
While PSRO offers a robust framework for finding equilibria in complex ABMs, it faces challenges regarding computational complexity as the policy pool grows. Future work will focus on scaling this framework to thousands of heterogeneous agents and incorporating more complex economic components like international trade and supply chains.
Summary updated