Incentive-Compatible Experimental Design

Panos Toulis, David C. Parkes, Elery Pfeffer, James Y. Zou

Introduction

In classical experimental design, treatments are typically viewed as passive entities—a chemical compound, a specific UI color, or a dosage of fertilizer. However, in modern digital ecosystems, treatments are often administered by strategic, self-interested agents. Consider a company hiring two competing advertising agencies to run a pilot campaign. These agencies are not passive; they are motivated to win the long-term contract. This introduces a layer of game theory into statistical design: if the evaluation metric is poorly chosen, agents may "game" the experiment by adopting high-risk strategies rather than performing their best "natural" action.

This paper introduces a framework for incentive-compatible experimental design. By bridging non-cooperative game theory with variance-stabilizing transformations, the authors demonstrate how to construct experiments where an agent’s dominant strategy is to act naturally, ensuring that the experimenter identifies the truly superior performer.

The Challenge of Strategic Agents

The presence of strategic agents introduces two primary complications that traditional A/B testing is ill-equipped to handle:

  1. Strategic Behavior: If an agent is evaluated solely on their average performance, a lower-quality agent might intentionally choose a high-variance strategy. Much like a trailing sports team taking "Hail Mary" passes, the agent hopes for a lucky outlier that allows them to surpass a more consistent, higher-quality competitor.
  2. Strategic Interference: In networked environments, such as viral marketing or social platforms, the actions of one agent can influence the outcomes of another. This "interference" (e.g., free-riding on a competitor's brand awareness) makes it difficult to isolate individual performance and creates incentives for agents to exploit the efforts of others.

The goal of an incentive-compatible design is to ensure that the agent’s "natural action"—the action they would take to maximize performance in a non-competitive, isolated setting—is also the action that maximizes their probability of winning the experiment.

Methodology: Variance Stabilization and the Delta Method

The core technical contribution of the paper is the application of variance-stabilizing transformations to performance scores.

The "Highest Average" Trap

The authors argue that simply selecting the agent with the highest average performance is often not incentive-compatible. When performance follows certain distributions, higher means are often coupled with higher variances. If the experimenter uses a raw mean, the agent can manipulate the variance to inflate their winning probability.

The Delta Method Approach

To counter this, the authors utilize the Delta method and asymptotic theory to transform raw scores. By applying a specific function to the results—such as taking the square root of Poisson-distributed data or the negative reciprocal of certain Normal-distributed results—the experimenter can make the variance of the score independent of the agent's specific strategy choice.

When the variance is stabilized, the agent can no longer "game" the system by increasing risk. Their only path to winning is to maximize the expected value of the transformed score, which the authors align with the agent's natural, high-quality action.

Solving for Interference

To address strategic interference, the authors propose a novel experimental structure. Instead of a simple split, they divide experimental units into multiple groups ($G_1, G_2$) and distinct test sets. By observing how agents perform across these overlapping groups, the experimenter can set up a system of equations to solve for individual performance parameters, effectively "teasing out" the true quality of an agent even in the presence of free-riding or competitive spillover.

Key Results and Simulations

The paper validates these theoretical constructs through synthetic simulations based on Normal and Poisson distributions:

  • Poisson Marketing Simulation: In a simulated environment where agent performance followed a Poisson distribution, the authors compared a standard linear mean ($x$) against a square-root transformation ($\sqrt{x}$). With 50 units per block, the win rate for the truly superior agent increased from 85% to 91% simply by changing the scoring function.
  • Incentive Alignment: The mathematical proofs (Theorems 3.1 and 4.1) provide a "recipe" for constructing these score functions. They demonstrate that for a wide class of distributions, there exists a transformation that renders the experiment incentive-compatible.
  • Power vs. Fairness: Interestingly, the authors found that making an experiment incentive-compatible does not necessarily come at the cost of statistical power. In multi-block designs, variance stabilization actually improved the probability that the best agent would be correctly identified.

Limitations and Future Directions

While the framework provides a robust foundation for strategic experimental design, several challenges remain:

  • Parametric Dependency: The choice of the variance-stabilizing transformation currently requires knowledge of the underlying data distribution (e.g., knowing the data is Poisson).
  • Interference Parameters: The current model for resolving interference assumes the experimenter knows the degree of interference ($\gamma$), which may not be true in practice.
  • Asymptotic Assumptions: The proofs rely on large-sample theory; the behavior of these designs in small-scale or "thin-data" environments requires further investigation.

Future research may focus on non-parametric versions of these tests that do not require prior distributional knowledge, as well as methods to estimate interference as a "nuisance parameter" during the experimental process.

Conclusion

As experiments move from the lab to complex, agent-driven marketplaces, the assumptions of passive treatment must be discarded. This paper provides a critical bridge between statistics and game theory, offering a practical toolkit for designing experiments that are not only statistically sound but also resistant to strategic manipulation.

Summary updated

Abstract

We consider the design of experiments to evaluate treatments that are administered by self-interested agents, each seeking to achieve the highest evaluation and win the experiment. For example, in an advertising experiment, a company wishes to evaluate two marketing agents in terms of their efficacy in viral marketing, and assign a contract to the winner agent. Contrary to traditional experimental design, this problem has two new implications. First, the experiment induces a game among agents, where each agent can select from multiple versions of the treatment it administers. Second, the action of one agent -- selection of treatment version -- may affect the actions of another agent, with the resulting strategic interference complicating the evaluation of agents. An incentive-compatible experiment design is one with an equilibrium where each agent selects its natural action, which is the action that maximizes the performance of the agent without competition (e.g., expected number of conversions if agent is assigned the advertising contract). Under a general formulation of block experiment designs, we identify sufficient conditions that guarantee incentive-compatible experiments.These conditions rely on the existence of statistics that can estimate how agents would perform without competition,and their use in constructing score functions to evaluate the agents. In the setting with no strategic interference, we also study the power of the design, i.e., the probability that the best agent wins, and show how to improve the power of incentive-compatible designs.From the technical side, our theory uses a range of statistical methods such as hypothesis testing, variance-stabilizing transformations and the Delta method, all of which rely on asymptotics.
BibTeX
@inproceedings{toulis2015incentive,
  author = {Panos Toulis and David C. Parkes and Elery Pfeffer and James Y. Zou},
  title = {{Incentive-Compatible Experimental Design}},
  booktitle = {EC},
  year = {2015},
  doi = {10.1145/2764468.2764525},
}
Figure