Abstract
We develop a difference-in-differences framework to measure the persuasive impact of informational treatments on behavior. We introduce two causal parameters, the forward and backward average persuasion rates on the treated, which refine the average treatment effect on the treated. The forward rate excludes cases of "preaching to the converted," while the backward rate omits "talking to a brick wall" cases. We propose both regression-based and semiparametrically efficient estimators. The framework applies to both two-period and staggered treatment settings, including event studies, and we demonstrate its usefulness with applications to a British election and a Chinese curriculum reform.
Introduction
In fields ranging from political science to public health, researchers frequently seek to measure how effectively a message—such as a campaign ad or an educational program—changes behavior. The standard metric for such evaluations is the Average Treatment Effect on the Treated (ATT). However, the ATT can be a misleading indicator of a message's true persuasive power.
If a message is delivered to an audience that already agrees with it, the ATT will be small because there was little room for change—a phenomenon often described as "preaching to the converted." Conversely, if the audience is entirely resistant, the message hits a "brick wall." In both cases, the raw ATT dilutes the impact of the message among those who were actually persuadable. This paper bridges the gap between experimental persuasion literature and observational research by integrating persuasion metrics into the popular Difference-in-Differences (DiD) framework.
Defining Persuasion Rates
To provide a more granular view of causal impact, we introduce two primary parameters that refine the ATT:
- Forward Persuasion Rate (FPR): This measures the effect of the treatment specifically among those who would not have taken the action without the treatment. By removing the "already converted" from the denominator, the FPR isolates how effective the message was at moving the needle among those initially unswayed.
- Backward Persuasion Rate (BPR): This measures the effect among those who did take the action, asking what proportion of that behavior was caused specifically by the treatment. This effectively filters out the "brick wall" cases—those who would never have changed their minds regardless of the intervention.
These rates are conceptually linked to Judea Pearl’s causal framework: the FPR is analogous to the "Probability of Sufficiency," while the BPR aligns with the "Probability of Necessity."
Identification and Estimation
Identifying these rates in observational settings requires two foundational assumptions. First, we rely on the standard Parallel Trends assumption of DiD, which posits that the treated and control groups would have followed the same trajectory in the absence of treatment. Second, we introduce a No-Backlash (Monotonicity) assumption—the requirement that the message does not accidentally drive individuals to do the exact opposite of the intended behavior.
We propose several estimation strategies to implement this framework:
- Regression-based: Simple adjustments to the interaction terms in standard two-way fixed effects models.
- 2SLS/GMM: Reinterpreting the persuasion problem as an instrumental variable setup where the treatment acts as an instrument for a pseudo-outcome.
- Doubly Robust (DR): A semiparametrically efficient estimator that remains consistent even if either the propensity score or the outcome model is misspecified.
Our framework also extends to staggered adoption designs, allowing researchers to calculate "Event Study" persuasion rates when different groups receive the treatment at different times.
Empirical Applications
British Election Panel Study (1992–1997)
We applied our framework to analyze how shifts in newspaper endorsements influenced voting behavior. When several major UK newspapers switched their support from the Conservative to the Labour party between elections, the standard ATT of the switch was approximately 8.9%. However, our Forward Persuasion Rate (FPR) was ~17.6%. This suggests that among readers who were not already "converted" to Labour, the newspapers were nearly twice as effective as the raw ATT would suggest.
Chinese Curriculum Reform
We evaluated a staggered rollout of new high school textbooks in China designed to shift political attitudes. Using survey data from Peking University, we found that the new curriculum significantly increased trust in the central government. Our event-study persuasion rates were consistently higher than the original study's estimates, highlighting that the reform was highly effective among the previously unconvinced student population.
Technical Insights
A significant contribution of this research is identifying the "small denominator" problem. When attempting to measure persuasion in a group where very few people are persuadable, the math becomes sensitive, similar to the weak instrument problem in econometrics. We provide robust inference methods, including Anderson-Rubin statistics, to handle these cases. Additionally, we offer a "back-of-the-envelope" method that allows researchers to estimate these rates using only the final ATT and the share of "non-takers" from published studies, even without access to the original microdata.
Conclusion
By rescaling the ATT into Forward and Backward persuasion rates, this framework provides a "pure" measure of a message's power. While the current model focuses on binary outcomes and assumes a lack of backlash, it offers a significant advancement in how we interpret informational treatments in observational data, ensuring that we do not undervalue the impact of messages delivered to the already-convinced or the inherently resistant.
Summary updated