An Experimental Study of Recommendation Algorithms for Tailored Health Communication

Hyun Suk Kim, Sijia Yang, Minji Kim, Brett Hemenway, Lyle Ungar, Joseph N. Cappella

TL;DR

This paper explores using recommendation algorithms, common in e-commerce, to tailor health messages (specifically anti-smoking PSAs) to individual smokers. They found that their tailored algorithm predicted message persuasiveness more accurately than simpler methods and led to a greater likelihood of smokers evaluating PSAs favorably and attempting to quit smoking.

What Problem Does This Paper Address?

The effectiveness of health communication campaigns, particularly those aiming to change behavior like smoking cessation, often relies on tailoring messages to specific audiences. While message tailoring is known to enhance persuasion, traditional methods are resource-intensive. The authors investigate whether recommendation algorithms, successful in commercial settings for predicting user preferences, can be effectively adapted to predict individual responses to health messages and thereby improve campaign outcomes. They aim to bridge the gap between the proven effectiveness of tailoring and the practical challenges of implementing it at scale for complex health interventions.

Methods

The core of the study is the development and experimental testing of a message tailoring algorithm. This algorithm leverages a combination of:

  • Collaborative Filtering: User-based (UBCF) and item-based (IBCF) approaches to identify similar users and similar messages based on past ratings.
  • Matrix Factorization (MF): A more advanced technique to uncover latent features in user-item interactions.
  • Individual-Level Data: Information about the smoker, such as stage of change, nicotine dependence, and demographics.
  • Message-Level Data: Characteristics of the PSAs, including argument strength, narrative presence, and emotional content.
  • Interactions: Combinations of individual and message features to capture more nuanced preferences.

This algorithm was trained on data from Wave 1 of a longitudinal study and then applied in Wave 2 to predict participants' persuasiveness ratings for PSAs they hadn't seen. The algorithm's performance was compared against two baseline methods: "Off the Shelf" (OTS), which involved random selection of PSAs, and "Best in Show" (BIS), which selected PSAs most preferred by the general smoker population.

Datasets

The study utilized data collected from a three-wave online experiment.

  • Wave 1 (N=1,057): Baseline survey data, including demographic information, smoking history, nicotine dependence, and initial evaluations of eight PSAs. This data was used to train the tailoring algorithm.
  • Wave 2 (N=675): Participants were randomly assigned to one of three conditions (OTS, BIS, Tailoring) and viewed four PSAs. They then evaluated these PSAs and reported their smoking cessation intentions.
  • Wave 3 (N=525): A follow-up survey to assess self-reported quitting behavior.

The PSAs themselves were drawn from a larger archive of 200 anti-smoking PSAs, with 80 selected for this study based on varying levels of perceived effectiveness (PE). PE data for these PSAs was previously collected from independent samples of smokers.

Key Insights and Ideas

  • Algorithmic Tailoring Mimics Social Influence: The study suggests that recommendation algorithms, by leveraging collaborative filtering, can mimic social influence processes. They recommend messages that similar others find persuasive, but without direct social interaction, offering a novel way to influence behavior.
  • Behavioral Data is Key: The tailoring algorithm's success is attributed to its focus on actual choices and preferences (behavioral data) rather than solely relying on psychographic predictors, which is a departure from some traditional tailoring methods.
  • Predictive Accuracy Matters for Impact: The superior predictive accuracy of the tailoring algorithm, as measured by lower RMSE, directly translated into more favorable PSA evaluations and a higher likelihood of quitting attempts. This highlights the critical link between accurate prediction and behavioral outcomes.
  • Within-Participant Variability is Important: The tailoring algorithm's ability to reduce within-participant variability in PSA ratings suggests it better captures individual latent preferences, leading to more consistent and potentially more impactful recommendations.

Key Results

  • Predictive Accuracy: The tailoring algorithm achieved significantly lower Root Mean Square Error (RMSE) in predicting participants' perceived effectiveness (PE) ratings compared to the "Best in Show" (BIS) method (.54 vs. .88 on average).
  • Message Evaluation: Participants exposed to PSAs recommended by the tailoring algorithm evaluated them as significantly more persuasive than those in the "Off the Shelf" (OTS) condition. The difference between BIS and OTS was not statistically significant.
  • Behavioral Impact: Smokers who viewed PSAs recommended by the tailoring algorithm were more likely to evaluate the PSAs favorably and attempt to quit smoking compared to those receiving randomly selected PSAs (OTS).
  • No Significant Difference for BIS/OTS: The "Best in Show" (BIS) method did not show a significant advantage over the "Off the Shelf" (OTS) method in terms of message assessment or quitting behavior.

Limitations and Future Work

  • Data Intensity: The authors acknowledge that their methodology is data-intensive, which might pose a challenge for practical implementation. They suggest that micro-targeting smaller, homogeneous groups could reduce data burdens.
  • Algorithm Comparison: The study focused on comparing their developed tailoring algorithm against two standard approaches, rather than comparing different types of tailoring algorithms against each other.
  • Small Effects: While statistically significant, the observed behavioral effects (quit attempts) were small. The authors note that this is understandable given the difficulty of changing established behaviors like smoking, especially among those who haven't yet quit.
  • Future Research: The paper suggests exploring alternative collaborative filtering approaches beyond those tested and investigating the effectiveness of micro-targeting. Further research is also needed to explore how recommendation systems can be optimized for various health communication contexts and to understand the nuances of how these algorithms influence behavior.

Summary updated

Abstract

Recommendation algorithms are widely used in online cultural markets to provide personalized suggestions for products like books and movies. At the heart of the commercial success of recommendation algorithms is their ability to make an accurate prediction of a target person’s preferences for previously unseen items. Can these algorithms also be used to predict which health messages an individual will evaluate favorably, and thereby provide effective tailored communication to the person? Although there is evidence that message tailoring enhances persuasion, little research has examined the effectiveness of recommendation algorithms for tailored health interventions aimed at promoting behavior change. We developed a message tailoring algorithm to select smoking-related public service announcements (PSAs) for smokers, and experimentally test its effectiveness in predicting a target smoker’s evaluations of PSAs and encouraging smoking cessation. The tailoring algorithm was constructed using multiple levels of data on smokers’ PSA rating history, individual differences, content features of the PSAs, and other smokers’ PSA ratings. We conducted a longitudinal online experiment to examine its efficacy in comparison to two non-tailored methods: “best in show” (choosing messages most preferred by other smokers) and “off the shelf” (random selection from eligible ads). The results showed that the tailoring algorithm produced more accurate predictions of smokers’ message evaluations than the simple-average method used for the “best in show” approach. Smokers who viewed PSAs recommended by the tailoring algorithm were more likely than those receiving a random set to evaluate the PSAs favorably and quit smoking. There was no significant difference between the “best in show” and “off the shelf” methods in message assessment and quitting behavior.
BibTeX
@inproceedings{kim2019experimental,
  author = {Hyun Suk Kim and Sijia Yang and Minji Kim and Brett Hemenway and Lyle Ungar and Joseph N. Cappella},
  title = {{An Experimental Study of Recommendation Algorithms for Tailored Health Communication}},
  booktitle = {Computational},
  year = {2019},
  doi = {10.5117/ccr2019.1.005.sukk},
}
Figure