Introduction
Predicting how strategic agents will react to changes in market rules is a fundamental challenge in economic engineering. Whether designing ad auctions, ride-sharing platforms, or school assignment systems, analysts must answer the "counterfactual question": If we change the rules, how will the outcome change? Traditional structural estimation methods often provide precise point estimates, but these predictions are notoriously fragile. This paper introduces Robust Multi-agent Counterfactual Prediction (RMAC), a framework that accounts for small deviations in rationality and model accuracy to provide robust "best-case" and "worst-case" bounds on outcomes.
The Challenge of Counterfactuals
Standard approaches to counterfactual prediction rely on three demanding assumptions:
- Perfect Rationality: Agents are assumed to perfectly optimize their utility.
- Correct Specification: The analyst's model of the environment and agent utilities is assumed to be 100% accurate.
- Identification: Observed actions are assumed to map uniquely to underlying agent preferences ("types").
In practice, these assumptions are frequently violated. Humans exhibit bounded rationality, models are mere approximations, and different preferences can often lead to identical observed behaviors. When these assumptions fail even slightly, standard point-estimate predictions can become wildly inaccurate.
The RMAC Framework
RMAC shifts the focus from finding a single "correct" model to identifying the set of all plausible behaviors consistent with the data.
The framework is built on several key innovations:
- The Revelation Game: The estimation problem is reframed as a game where "data-players" report types and counterfactual actions to minimize regret in both the original and new environments.
- $ε$-Equilibrium: Instead of a single equilibrium, RMAC considers the set of all behaviors where agents have at most a small amount ($ε$) of regret. This $ε$ parameter explicitly models the degree of allowed irrationality or model misspecification.
- Robustness Intervals: By calculating the maximum and minimum values of an evaluation function (e.g., revenue or welfare) within the $ε$-set, RMAC provides a robust interval rather than a single point.
- Revelation Game Fictitious Play (RFP): To handle the computational complexity of finding these bounds, the authors use RFP, a first-order optimization algorithm inspired by game-theoretic learning.
Experimental Results
The authors evaluated RMAC across several classic economic environments, including auctions, school choice, and social choice.
Auction Sensitivity
A key finding is the extreme sensitivity of auction revenue to small deviations in rationality. In simulations of first-price and second-price auctions, even a tiny $ε$ (e.g., 0.01) leads to significantly wider revenue bounds compared to standard statistical error bars.
As shown in the figures, as the allowed deviation $ε$ increases from 0 to 0.01, the uncertainty regarding counterfactual revenue grows substantially. This suggests that point-estimate predictions in auction design may be riskier than previously assumed. RMAC also naturally handles "censored" data—where agent types are not fully identified—by producing wider bounds in those regions.
Mechanism Comparison
In social choice settings, RMAC reveals that some mechanisms are inherently more robust than others. For instance, the VCG mechanism proved more robust than the Median mechanism. In VCG, any deviation from the equilibrium strategy directly impacts the price an agent pays, making it harder for behavior to "drift" without incurring significant regret.
Conclusion
RMAC provides a principled way to quantify the uncertainty inherent in multi-agent counterfactual predictions. By moving away from fragile point estimates and toward robust intervals, it offers a more reliable tool for market designers. While finding global bounds remains computationally challenging (NP-hard), the RFP algorithm provides a practical means of approximation. Future work aims to scale these methods to continuous action spaces and incorporate more sophisticated behavioral models to further refine the robustness intervals.
Summary updated