Abstract
Planned experiments are the gold standard for reliably comparing the causal effect of switching from a baseline policy to a new policy. However, a critical shortcoming of classical experimental methods is that they typically do not account for the dynamic nature of responses to policy changes. In multiagent economies, such as ad auctions, agents adapt their bidding strategies in response to experimental pricing changes. Consequently, the causal effects after this adaptation period—the long-term causal effects—are not captured by classical methodology. We formalize a framework to define and estimate these effects by leveraging behavioral game theory to formulate necessary ignorability assumptions. Our approach combines a behavioral model of agent actions with a temporal model of behavior evolution.
Introduction
In modern digital economies like online ad auctions or ride-sharing platforms, policy changes rarely result in immediate, stable outcomes. Instead, they trigger a chain of adaptations as agents learn and optimize their strategies. Standard A/B testing, while effective for static environments, often fails here because it captures only the "transient" state immediately following a change.
The challenge is twofold. First, the Fundamental Problem of Causal Inference prevents us from observing the same agent under both treatment and control conditions simultaneously. Second, the Dynamical Problem means that short-term data does not represent the long-term steady-state value of a policy. Traditional econometric tools like Difference-in-Differences (DID) often rely on linear assumptions that break down when agents interact strategically.
The Latent Space Framework
To address these challenges, we propose a latent space approach that bridges the gap between short-term observations and long-term equilibrium.
The framework relies on three core components:
- Behavioral Model: Instead of modeling raw actions, we model latent "behaviors" using Quantal Response models from behavioral game theory. This allows us to account for varying levels of strategic sophistication among agents.
- Temporal Model: A Vector Autoregressive (VAR(1)) model describes how these latent behaviors evolve over time as agents learn.
- The Pivot Assumption: We assume that at the start of an experiment ($t=0$), the distribution of behaviors is identical across groups because agents have not yet had time to react to their assignments. This "pivot" point allows us to extrapolate potential outcomes.
By formulating a "behavioral ignorability" assumption, we argue that while specific actions depend on experimental assignment, the underlying process of behavior evolution remains stable, enabling valid causal inference in the latent space.
Experimental Results
We validated this framework using the Rapoport and Boebel (1992) dataset, which involves human subjects playing zero-sum games. By treating different payoff matrices as distinct "policies" and holding out final periods of play as ground truth, we compared our Long-term Average Causal Effect (LACE) estimator against Naive and DID approaches.
The results demonstrated a significant improvement in accuracy. The LACE estimator achieved a Mean Squared Error (MSE) of 0.045, compared to 0.185 for the Naive approach and 0.361 for DID. This represents a 4x to 8x reduction in error, highlighting the value of injecting game-theoretic structure into the statistical model.
Discussion and Limitations
The primary insight of this work is that game theory can serve as a structural bridge for causal inference in complex systems. By modeling how agents should respond, we can make informed predictions about unobserved long-term states.
However, several limitations remain. The current model assumes a finite set of possible behaviors, and its accuracy depends heavily on the chosen game-theoretic model. Furthermore, the time horizon for "long-term" stabilization must be predefined. Future research will explore non-parametric behavior learning and more complex learning models, such as no-regret learning, to further refine these estimates.
Summary updated