A method for assessing the success and failure of community-level interventions in the presence of network diffusion, social reinforcement, and related social effects

Hsuan‐Wei Lee, G. Robin Gauthier, Jerreed Ivanich, Lisa Wexler, Bilal Khan, Kirk Dombrowski

Abstract

Prevention and intervention work done within community settings often face unique analytic challenges for rigorous evaluations. Since community prevention work (often geographically isolated) cannot be controlled in the same way other prevention programs and these communities have an increased level of interpersonal interactions, rigorous evaluations are needed. Even when the 'gold standard' randomized control trials are implemented within community intervention work, the threats to internal validity can be called into question given informal social spread of information in closed network settings. A new prevention evaluation method is presented here to disentangle the social influences assumed to influence prevention effects within communities. We formally introduce the method and its utility for a suicide prevention program implemented in several Alaska Native villages. The results show promise to explore eight sociological measures of intervention effects in the face of social diffusion, social reinforcement, and direct treatment. Policy and research implications are discussed.

Introduction

In public health, the Randomized Control Trial (RCT) is often considered the "gold standard" for evaluation. However, RCTs rely on the Stable Unit Treatment Value Assumption (SUTVA), which posits that the treatment of one individual does not affect the outcomes of others. In small, geographically isolated, or highly connected communities—such as Alaska Native villages—this assumption frequently fails. Social "contamination" occurs when participants in a treatment group share insights with those in a control group, potentially making an effective intervention appear unsuccessful in traditional analyses.

This paper introduces a novel framework for evaluating community-level interventions by analyzing social networks rather than just individual outcomes. By treating social ties as a feature of the intervention rather than a methodological hurdle, researchers can disentangle direct treatment effects from social diffusion and reinforcement, providing a clearer picture of how programs actually function in tight-knit settings.

Methodology

The proposed method is based on dyadic state transitions. Instead of tracking individuals in isolation, the model analyzes "ego-alter" pairs (a person and their social contact) across two time points ($t_0$ and $t_1$).

To categorize these interactions, the authors utilize a 6-bit counting function that tracks:

  1. Participation (2 bits): Did the ego participate? Did the alter participate?
  2. Initial Behavior (2 bits): Did the ego exhibit the protective behavior at $t_0$? Did the alter?
  3. Final Behavior (2 bits): Did they exhibit it at $t_1$?

This creates 64 possible dyad types. From these, the authors derive Eight Sociological Measures (M1–M8) to test specific hypotheses, including:

  • Direct Treatment/Prevention: Effectiveness for those who directly attended the program.
  • Social Effects: Whether an ego’s participation caused their alter to change.
  • Reinforcement: Whether the program works more effectively when friends attend together.
  • Diffusion: Whether the program "leaked" to non-participants through social ties.

Statistical significance is determined by comparing observed results against a Poisson-Binomial distribution generated via bootstrapping, allowing researchers to identify effects that exceed what would be expected by chance.

Data and Validation

The framework was validated using two primary datasets:

  1. Simulated Dataset: A synthetic network of 444 middle school children was used to confirm the mathematical model's accuracy against a known "ground truth."
  2. PC CARES (Field Data): Data from the "Promoting Community Conversations About Research to End Suicide" program, involving 447 individuals across six Alaska Native villages. Participants were surveyed on 39 specific protective behaviors, such as discussing gun safety or youth suicide risks.

Key Results and Insights

The evaluation of the PC CARES program yielded several significant findings:

  • Overall Success: 73% (66 out of 91) of the significant relationships identified across the 39 items were positive, indicating the program's broad effectiveness.
  • The Power of the Alter: The "Social Effect of Prevention" was the strongest area of impact. For 26 out of 39 items, having a social contact participate in the program was associated with the ego maintaining protective behaviors, even if the ego did not participate themselves.
  • Behavior-Specific Diffusion: Some behaviors spread more easily than others. Item 22 ("conversations about making it harder for an 'at risk' person to get a loaded gun") showed significant positive change across four of the eight sociological measures. Conversely, "diffusion of change" was less effective for reaching out to those who attempted suicide (Item 11), suggesting that some behaviors require direct training rather than informal social spread.

Discussion and Limitations

This method demonstrates that "contamination" is actually valuable information. If a program spreads through word-of-mouth, it is a sign of success. However, the authors acknowledge several hurdles to implementing this model:

  • Data Burden: The method requires high-quality relational data, necessitating the mapping of social networks alongside behavioral outcomes.
  • Sample Size: The mathematical model requires a certain "cell size" within the 64 dyad categories to remain statistically robust. In very small samples, some sociological measures may be impossible to calculate.
  • Measurement Timing: Seasonal behaviors may introduce bias if survey timing is not carefully controlled.

Conclusion

By shifting the focus from individuals to dyads, this framework allows researchers to measure the true reach of community interventions. It is particularly suited for "close-knit" contexts, such as school-wide programs or health initiatives in rural communities, where kinship and social ties are the primary drivers of behavioral change.

Summary updated

Abstract

Prevention and intervention work done within community settings often face unique analytic challenges for rigorous evaluations. Since community prevention work (often geographically isolated) cannot be controlled in the same way other prevention programs and these communities have an increased level of interpersonal interactions, rigorous evaluations are needed. Even when the `gold standard' randomized control trials are implemented within community intervention work, the threats to internal validity can be called into question given informal social spread of information in closed network settings. A new prevention evaluation method is presented here to disentangle the social influences assumed to influence prevention effects within communities. We formally introduce the method and it's utility for a suicide prevention program implemented in several Alaska Native villages. The results show promise to explore eight sociological measures of intervention effects in the face of social diffusion, social reinforcement, and direct treatment. Policy and research implication are discussed.
BibTeX
@misc{lee2018method,
  author = {Hsuan‐Wei Lee and G. Robin Gauthier and Jerreed Ivanich and Lisa Wexler and Bilal Khan and Kirk Dombrowski},
  title = {{A method for assessing the success and failure of community-level interventions in the presence of network diffusion, social reinforcement, and related social effects}},
  year = {2018},
  eprint = {1801.08612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1801.08612},
}
Figure