Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

Yucheng Yang, Chiyuan Wang, Andreas Schaab, Benjamin MollarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces Structural Reinforcement Learning (SRL), a novel method for solving complex macroeconomic models with many interacting agents. By having agents learn equilibrium prices from simulations and leverage their knowledge of individual dynamics, SRL bypasses computationally intensive "Master equations" and solves challenging models in minutes, a significant speedup over traditional methods.

What Problem Does This Paper Address?

Macroeconomic models with heterogeneous agents (individuals or firms with different characteristics) and aggregate risk (shocks affecting the whole economy) are crucial for understanding phenomena like inequality and business cycles. A major computational hurdle in these models is the "Master equation," which describes the evolution of the cross-sectional distribution of agents. This distribution becomes a high-dimensional state variable, leading to a "curse of dimensionality" that makes finding global solutions computationally intractable for many realistic models, especially those with complex market-clearing conditions.

Methods

The authors propose a Structural Reinforcement Learning (SRL) approach. Instead of directly solving the Master equation, SRL treats equilibrium prices as low-dimensional state variables. Agents learn optimal behavior by:

  1. Simulating Equilibrium Paths: Agents observe simulated price histories and learn from them.
  2. Exploiting Structural Knowledge: Crucially, agents know their own individual dynamics (e.g., how their wealth or income changes based on their decisions). This "structural" aspect allows for exact policy gradients, unlike standard RL which often relies on approximations.
  3. Structural Policy Gradient (SPG) Algorithm: This algorithm combines the RL principle of learning from simulations with the agents' structural knowledge to efficiently compute optimal policies.
  4. Restricted Perceptions Equilibrium (RPE): Agents' policies depend only on current prices (or a short price history), simplifying their decision problem and keeping the state space manageable. This is a variant of rational expectations where agents' beliefs are consistent with equilibrium outcomes but might not fully grasp the complex price formation process.

Datasets

Not discussed in the paper. The approach relies on simulated data generated by the models themselves.

Key Insights and Ideas

  • Sidestepping the Master Equation: The core innovation is bypassing the high-dimensional Master equation by focusing on low-dimensional prices as the agent's observable state. This is a major computational breakthrough.
  • "Structural" RL: Leveraging agents' knowledge of their own dynamics (e.g., budget constraints) allows for exact policy gradients, making the learning process more efficient and accurate than model-free RL.
  • Prices as Sufficient Statistics: The assumption that agents condition only on current prices (or a short history) simplifies the problem significantly, yet the results show this is often sufficient for capturing key economic dynamics.
  • Efficient Market Clearing: The method efficiently handles non-trivial market-clearing conditions by treating policy functions as supply schedules, which can be aggregated to find equilibrium prices.

Key Results

  • Speed: The SRL method solves benchmark heterogeneous agent models (Huggett, Krusell-Smith, HANK) globally within minutes, a dramatic improvement over traditional methods that can take hours or days.
  • Krusell-Smith: ~55 seconds
    • Huggett with aggregate shocks: ~75 seconds
    • HANK with forward-looking Phillips curve: ~3 minutes
  • Accuracy: The solutions obtained are highly accurate, closely matching rational expectations solutions from established methods (e.g., DeepHAM for Krusell-Smith) and value function iteration in simpler settings.
  • Robustness: The method is robust to the number of simulated trajectories used for training, with confidence bands shrinking as more data is used, indicating stable learning.
  • Information Sufficiency: In the Huggett model, conditioning on lagged prices provided minimal additional information for optimal consumption decisions, suggesting current prices already summarize relevant historical information.

Limitations and Future Work

  • Empirical Realism: The authors acknowledge that SRL is primarily a computational tool, not necessarily a realistic model of human learning.
  • Unbiased Price Expectations: The current approach assumes agents sample equilibrium prices without bias. Future work could relax this to incorporate empirically observed biases in price expectation formation (e.g., overweighting personal experiences).
  • Model-Free vs. Model-Based RL: The current approach is model-free regarding prices. Future work could explore model-based RL for price expectations.
  • Online Learning: Developing a fully online, incremental RL algorithm where agents update policies continuously while interacting with the environment is a promising direction.

Summary updated

Abstract

We present a new approach to formulating and solving heterogeneous agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let agents learn equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting agents' structural knowledge of their own individual dynamics. Our SRL method yields a general and highly efficient global solution method for heterogeneous agent models that sidesteps the Master equation and handles problems traditional methods struggle with, in particular nontrivial market-clearing conditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which we solve globally within minutes.
BibTeX
@misc{yang2025structural,
  author = {Yucheng Yang and Chiyuan Wang and Andreas Schaab and Benjamin Moll},
  title = {{Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics}},
  year = {2025},
  eprint = {2512.18892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18892},
}

Loading related content...

Figure