This paper, authored by a team from Stanford, Harvard, and Meta, provides the first large-scale empirical look at whether social media advertising actually works for public health. By aggregating hundreds of randomized controlled trials, the researchers demonstrate that digital interventions are a potent, cost-effective tool for shifting public opinion during a global crisis.
TL;DR
This study analyzes 819 randomized experiments involving 2.1 billion people to show that social media ad campaigns effectively shifted COVID-19 attitudes by approximately 1% at a cost of just $3.41 per influenced person. The results suggest that while individual ads have small effects, their massive scalability makes them a highly efficient tool for information dissemination and public health nudging.
What Problem Does This Paper Address?
During the COVID-19 pandemic, public health organizations spent tens of millions of dollars on social media ads to encourage vaccination and masking. However, measuring the actual "lift" of these ads is notoriously difficult.
Individual advertising experiments are often "underpowered"—meaning the sample size is too small to statistically detect the tiny changes (often <1%) that an ad might cause in a person's beliefs. Furthermore, non-experimental data is plagued by selection bias; people who see health ads might already be the type of people who seek out health information. This paper addresses these gaps by conducting a massive meta-analysis of randomized experiments to find the "true" average effect of digital health interventions.
Methods
The researchers utilized Meta’s "Brand Lift" infrastructure to conduct a meta-analysis of 819 randomized experiments.
- Randomization: Users were hashed into treatment and control groups. When a treatment user won an ad auction, they saw the public health ad. Control users were "held back" from seeing that specific ad.
- Surveying: Both groups were sent surveys 4–48 hours after the potential exposure.
- Statistical Modeling: The authors used a weighted linear model to estimate the treatment effect for each experiment, adjusting for demographics (age, gender) and platform-predicted engagement rates.
- Aggregation: They pooled these individual estimates using inverse variance weighting, a standard meta-analytic technique that gives more weight to experiments with more precise results (usually those with larger sample sizes).
Datasets
The study relies on a massive, unique dataset from Meta Platforms (Facebook and Instagram) spanning December 2020 to November 2021:
- Scope: 174 public health organizations (including ministries of health and NGOs).
- Scale: 376 distinct campaigns reaching 2.1 billion unique users.
- Responses: 1.5 million total survey responses.
- Outcomes: Responses were categorized into seven standardized metrics: Willingness (to get vaccinated), Importance, Effectiveness, Safety, Trustworthy Source, Knowledgeable (knowing where to get a vaccine), and Social Norms.
Key Insights and Ideas
- The "Knowledge" Advantage: The most significant finding was that ads are much better at informing than persuading. The "Knowledge" metric (knowing where to go for a vaccine) saw the largest lift. This suggests digital ads are a "particularly cost-effective channel for information dissemination" rather than just changing deep-seated beliefs.
- Small Effects, Massive Impact: On a per-person basis, the effect is tiny (a 0.55 percentage point shift). However, because the reach is in the billions, this translates to millions of people influenced.
- The Power of Pooling: The paper highlights that 90% of the individual experiments failed to reject the null hypothesis on their own. It was only by pooling the data that the researchers could "see" the signal through the noise, proving that many "failed" campaigns were actually working at a small, aggregate scale.
- Lower-Funnel Friction: Outcomes closer to actual behavior (like "Willingness" to get the shot) showed smaller, non-significant effects compared to "upper-funnel" outcomes like "Importance" or "Knowledge."
Key Results
- Average Treatment Effect: An increase of 0.55 percentage points in positive responses across all experiments (p = 2e-13).
- Aggregate Influence: An estimated 11.6 million individuals were influenced by these campaigns.
- Cost-Effectiveness: The average cost per influenced person was $3.41**. For the "Knowledge" metric specifically, the cost was even lower, at **$0.77 per person.
- Baseline Comparison: The 0.55 point shift represents roughly a 1% improvement over the baseline positive response rate of 55.7%.
Limitations and Future Work
- Beliefs vs. Action: The study measures self-reported attitudes and knowledge via surveys, not actual clinical outcomes (like shots in arms). Privacy and legal constraints prevent linking ad exposure directly to medical records.
- Survey Bias: There is a potential for "differential survey response"—the idea that people influenced by an ad might be more likely to answer a survey about that topic, though the authors run several robustness checks to mitigate this concern.
- Persuasion Decay: The study measures beliefs shortly after ad exposure (4–48 hours). It does not address how long these shifts in attitude last.
- Creative Heterogeneity: While the paper identifies "best practices" (e.g., using vertical video, credible messengers like faith leaders), it notes that even with this massive dataset, they are underpowered to conduct a rigorous meta-analysis on which specific creative elements (colors, fonts, specific wording) drive the most lift.
Summary updated