TL;DR
This paper introduces Structural Reinforcement Learning (SRL), a novel method for solving complex macroeconomic models with many interacting agents. By having agents learn equilibrium prices from simulations and leverage their knowledge of individual dynamics, SRL bypasses computationally intensive "Master equations" and solves challenging models in minutes, a significant speedup over traditional methods.
What Problem Does This Paper Address?
Macroeconomic models with heterogeneous agents (individuals or firms with different characteristics) and aggregate risk (shocks affecting the whole economy) are crucial for understanding phenomena like inequality and business cycles. A major computational hurdle in these models is the "Master equation," which describes the evolution of the cross-sectional distribution of agents. This distribution becomes a high-dimensional state variable, leading to a "curse of dimensionality" that makes finding global solutions computationally intractable for many realistic models, especially those with complex market-clearing conditions.
Methods
The authors propose a Structural Reinforcement Learning (SRL) approach. Instead of directly solving the Master equation, SRL treats equilibrium prices as low-dimensional state variables. Agents learn optimal behavior by:
- Simulating Equilibrium Paths: Agents observe simulated price histories and learn from them.
- Exploiting Structural Knowledge: Crucially, agents know their own individual dynamics (e.g., how their wealth or income changes based on their decisions). This "structural" aspect allows for exact policy gradients, unlike standard RL which often relies on approximations.
- Structural Policy Gradient (SPG) Algorithm: This algorithm combines the RL principle of learning from simulations with the agents' structural knowledge to efficiently compute optimal policies.
- Restricted Perceptions Equilibrium (RPE): Agents' policies depend only on current prices (or a short price history), simplifying their decision problem and keeping the state space manageable. This is a variant of rational expectations where agents' beliefs are consistent with equilibrium outcomes but might not fully grasp the complex price formation process.
Datasets
Not discussed in the paper. The approach relies on simulated data generated by the models themselves.
Key Insights and Ideas
- Sidestepping the Master Equation: The core innovation is bypassing the high-dimensional Master equation by focusing on low-dimensional prices as the agent's observable state. This is a major computational breakthrough.
- "Structural" RL: Leveraging agents' knowledge of their own dynamics (e.g., budget constraints) allows for exact policy gradients, making the learning process more efficient and accurate than model-free RL.
- Prices as Sufficient Statistics: The assumption that agents condition only on current prices (or a short history) simplifies the problem significantly, yet the results show this is often sufficient for capturing key economic dynamics.
- Efficient Market Clearing: The method efficiently handles non-trivial market-clearing conditions by treating policy functions as supply schedules, which can be aggregated to find equilibrium prices.
Key Results
- Speed: The SRL method solves benchmark heterogeneous agent models (Huggett, Krusell-Smith, HANK) globally within minutes, a dramatic improvement over traditional methods that can take hours or days.
- Krusell-Smith: ~55 seconds
- Huggett with aggregate shocks: ~75 seconds
- HANK with forward-looking Phillips curve: ~3 minutes
- Accuracy: The solutions obtained are highly accurate, closely matching rational expectations solutions from established methods (e.g., DeepHAM for Krusell-Smith) and value function iteration in simpler settings.
- Robustness: The method is robust to the number of simulated trajectories used for training, with confidence bands shrinking as more data is used, indicating stable learning.
- Information Sufficiency: In the Huggett model, conditioning on lagged prices provided minimal additional information for optimal consumption decisions, suggesting current prices already summarize relevant historical information.
Limitations and Future Work
- Empirical Realism: The authors acknowledge that SRL is primarily a computational tool, not necessarily a realistic model of human learning.
- Unbiased Price Expectations: The current approach assumes agents sample equilibrium prices without bias. Future work could relax this to incorporate empirically observed biases in price expectation formation (e.g., overweighting personal experiences).
- Model-Free vs. Model-Based RL: The current approach is model-free regarding prices. Future work could explore model-based RL for price expectations.
- Online Learning: Developing a fully online, incremental RL algorithm where agents update policies continuously while interacting with the environment is a promising direction.
Summary updated