Identifying the Effect of Persuasion

Sung Jae Jun, Sokbae Lee

TL;DR

This paper tackles the challenge of accurately measuring "persuasion" in economic contexts, arguing that common methods often overstate effects and mask important heterogeneity. The authors propose a more rigorous framework using potential outcomes and instrumental variables, offering new definitions and identification strategies that provide sharper, more reliable estimates of persuasion's true impact.

What Problem Does This Paper Address?

Economists are keen to understand how persuasive messages influence behavior – from voting to purchasing decisions. A widely used measure, often called the "persuasion rate," has been employed in numerous studies. However, its precise interpretation has been unclear, leading to potential misinterpretations and misleading comparisons across studies. The authors argue that this common measure, derived from a specific statistical approach, doesn't accurately capture the causal effect of persuasion on the general population. It can be biased, especially when individuals have different responses to persuasive messages (heterogeneity) or when the "treatment" (e.g., exposure to a persuasive message) is not perfectly observed. This ambiguity makes it difficult to draw reliable conclusions about the effectiveness of persuasive efforts.

Methods

The core of the paper lies in its rigorous econometric framework. The authors leverage the potential outcomes model, a standard tool in causal inference, to formally define the "persuasion rate" as a conditional probability: the likelihood of taking an action because of a persuasive message, given that the action wouldn't have been taken otherwise. They then explore how to identify this rate from data under various realistic scenarios of data availability, particularly when the actual exposure to the persuasive message (the treatment) is not perfectly observed.

Key methodological contributions include:

  • Formal Definition of Persuasion Rate: Moving beyond ad-hoc measures, they define it as $P\{Y_i(1) = 1 | Y_i(0) = 0\}$, where $Y_i(t)$ is the potential outcome for individual $i$ under treatment $t$.
  • Instrumental Variables (IV) Approach: They use binary instruments (like the availability of a cable channel) to address the endogeneity of treatment exposure.
  • Partial Identification: Recognizing that precise point identification is often impossible with real-world data, they derive sharp identified intervals (bounds) for the persuasion rate under different data assumptions. This acknowledges uncertainty and provides a more robust understanding of the possible range of effects.
  • Distinguishing Population vs. Subgroup Effects: They differentiate between the average persuasion rate for the entire population ($\theta_{avg}$) and the persuasion rate for "compliers" ($\theta_{local}$), those who would only be persuaded if exposed to the message.

Datasets

The paper uses existing datasets from several published studies to illustrate their methods and findings. These include:

  • DellaVigna and Kaplan (2007): Data on Fox News availability and voting behavior.
  • Gerber, Karlan, and Bergan (2009): Data from a field experiment on newspaper subscriptions and voting.
  • Chen and Yang (2019): Data from a field experiment on access to uncensored media in China.
  • Enikolopov, Petrova, and Zhuravskaya (2011): Data on NTV exposure and voting behavior in Russia.
  • Landry, Lange, List, Price, and Rupp (2006) & DellaVigna, List, and Malmendier (2012): Data from field experiments on door-to-door fundraising.

No new datasets are introduced.

Key Insights and Ideas

  • The "Common" Persuasion Measure is Flawed: The paper's central insight is that the widely used measure of persuasion often fails to estimate the true causal rate, frequently overstating the effect and masking crucial heterogeneity in how people respond. This means many previous studies might have drawn incorrect conclusions.
  • Compliers Matter, But So Does Everyone Else: While focusing on "compliers" (those who only act if persuaded) is informative, understanding the persuasion effect on the entire population is crucial for policy. The paper provides tools to estimate both.
  • Data Availability Dictates Identification: The ability to precisely identify persuasion effects heavily depends on what data is available (e.g., observing the actual treatment exposure vs. only the instrument). The paper systematically analyzes these different scenarios.
  • Continuous Instruments Unlock More Information: The presence of a continuous instrument (like signal strength) offers a much richer possibility for point identification, especially for understanding persuasion's effects across different segments of the population.

Key Results

  • DK's Measure is Misleading: The paper demonstrates through theoretical analysis and empirical examples that the commonly used measure (referred to as $\tilde{f}$ or $\tilde{f}_{DK}$) often yields estimates that are not interpretable as causal probabilities and can even exceed 1.
  • Partial Identification is the Norm: In most realistic scenarios, the persuasion rate is only partially identified, meaning researchers can only establish a range (an interval) within which the true effect lies. The paper provides sharp bounds for these intervals.
  • Heterogeneity is Pervasive: Empirical applications consistently show that persuasive effects are highly heterogeneous across individuals. For instance, in the uncensored media study, effects were large for some but negligible for others.
  • Lower Bounds are More Robust: The derived lower bounds ($\theta_L$) for average persuasion are often more reliable and informative than the upper bounds, especially when treatment exposure is not perfectly observed.
  • Local Persuasion Rates Can Be High: In some cases, the persuasion effect for compliers ($\theta_{local}$) can be substantial, even when the average population effect is less clear.

Limitations and Future Work

  • Focus on Binary Outcomes/Treatments: The primary analysis focuses on binary outcomes and treatments. While extensions to non-binary outcomes are discussed in the appendices, this remains a core limitation.
  • Assumptions: The identification results rely on specific assumptions, such as the absence of "defiers" in the IV framework (Assumption B) and monotonic treatment response (Assumption A). The paper discusses the implications of these assumptions.
  • Data Requirements: Achieving point identification often requires rich data, including the joint distribution of outcomes, treatment, and instruments, which may not always be available.
  • Future Research: The authors suggest extending the framework to multivalued treatments (e.g., different levels of exposure or different types of media) and exploring more complex structural models of persuasion. They also highlight the need for robust inference methods that account for partial identification.

Summary updated

Abstract

This paper examines a commonly used measure of persuasion whose precise interpretation has been obscure in the literature. By using the potential outcome framework, we define the causal persuasion rate by a proper conditional probability of taking the action of interest with a persuasive message conditional on not taking the action without the message. We then formally study identification under empirically relevant data scenarios and show that the commonly adopted measure generally does not estimate, but often overstates, the causal rate of persuasion. We discuss several new parameters of interest and provide practical methods for causal inference.
BibTeX
@article{jun2023identifying,
  author = {Sung Jae Jun and Sokbae Lee},
  title = {{Identifying the Effect of Persuasion}},
  journal = {Journal of Political Economy},
  year = {2023},
  doi = {10.1086/724114},
}
Figure