An Experimental Study of Recommendation Algorithms for Tailored Health Communication

hyun Suk Kim, Sijia Yang, Minji Kim, Brett Hemenway, Lyle Ungar, Joseph Nicholas Cappella

TL;DR

This paper explores using recommendation algorithms, common in e-commerce, to personalize health messages for smokers. They found that a tailored recommendation approach led to more accurate predictions of message persuasiveness and, crucially, increased smoking cessation attempts compared to random selection or "best in show" methods. This suggests a promising avenue for more effective public health interventions.

What Problem Does This Paper Address?

Personalized health communication, or "message tailoring," is known to be more persuasive than generic messages. However, creating tailored messages is complex, costly, and often relies on broad psychographic profiles. The authors investigate whether recommendation algorithms, which excel at predicting user preferences for unseen items in commercial settings (like Netflix or Amazon), can be adapted to predict which health messages (specifically, anti-smoking Public Service Announcements or PSAs) an individual smoker will find most persuasive. The core challenge is to bridge the gap between commercial recommendation systems and the nuanced domain of health behavior change, demonstrating their effectiveness beyond mere prediction to actual behavioral outcomes.

Methods

The researchers developed a message tailoring algorithm that leverages collaborative filtering techniques (both user-based and item-based) and matrix factorization. This algorithm incorporates:

  • Smokers' PSA rating history: How individuals rated previous PSAs.
  • Individual differences: Characteristics of the smoker (e.g., stage of change, nicotine dependence).
  • Content features of PSAs: Attributes of the messages themselves (e.g., argument strength, presence of narrative, smoking cues).
  • Other smokers' PSA ratings: Information from a larger group of smokers.

They compared this tailored approach against two simpler methods:

  1. "Off the Shelf" (OTS): Randomly selecting PSAs from a pool of eligible messages.
  2. "Best in Show" (BIS): Selecting PSAs that were most preferred by other smokers (using a simple average of ratings).

The study involved a longitudinal experiment with three waves: a baseline survey, a main experiment where participants viewed PSAs, and a follow-up survey to assess behavioral outcomes.

Datasets

The study utilized data from two main sources:

  • Wave 1 Data (N=1,057 smokers): This baseline survey provided individual characteristics and initial PSA evaluations. This data was used to train and develop the tailoring algorithm.
  • Wave 2 Data (N=675 participants): This was the main experimental phase where participants were randomly assigned to one of the three conditions (OTS, BIS, Tailoring) and viewed PSAs.
  • Wave 3 Data (N=525 participants): A follow-up survey to assess behavioral outcomes (smoking cessation attempts).

Additionally, a larger dataset of 200 anti-smoking PSAs, coded for various features and rated by independent samples of smokers, was used to inform the development of the tailoring algorithm and select the PSAs for the experiment.

Key Insights and Ideas

  • Recommendation algorithms can predict health message persuasiveness: The core idea is that the predictive power of recommendation systems, proven in commercial contexts, can be successfully applied to health communication.
  • Tailoring matters for behavior change: The study moves beyond just predicting message preference to demonstrating that tailored messages can influence actual behavior (quitting attempts).
  • "Best in Show" isn't always best: The "best in show" approach, while intuitive, did not significantly outperform random selection, suggesting that simply picking the most popular messages might not be effective for individual tailoring.
  • Within-participant variability is key: The tailoring algorithm reduced within-participant variability in PSA ratings, indicating it better approximated individual preferences compared to non-tailored methods. This suggests the algorithm is truly personalizing the experience.
  • Social influence without direct interaction: Collaborative filtering, a core component of the algorithm, mimics social influence by recommending what similar others liked, but does so without direct social interaction, offering a scalable approach.

Key Results

  • Predictive Accuracy: The tailoring algorithm demonstrated significantly higher predictive accuracy (lower RMSE) in predicting smokers' perceived effectiveness (PE) of PSAs compared to the simple-average "best in show" method, both in Wave 1 and Wave 2 data.
  • Attitudinal Effects: Smokers exposed to PSAs recommended by the tailoring algorithm evaluated them as significantly more persuasive than those who viewed OTS PSAs.
  • Behavioral Effects: Participants who received PSAs recommended by the tailoring algorithm were more likely to report quitting smoking than those who received randomly selected PSAs. This effect was mediated by increased perceived effectiveness and quitting intention.
  • No significant difference for BIS: The "best in show" method showed no significant difference in message assessment or quitting behavior compared to the "off the shelf" (random) method.

Limitations and Future Work

  • Data Intensity: The authors acknowledge that their methodology is data-heavy, which might be a concern for practical applications. However, they argue that other tailoring methods are also data-intensive. They suggest micro-targeting very small, homogeneous groups could reduce data burdens.
  • Algorithm Comparison: The study focused on comparing their tailored algorithm to two simpler methods, not on comparing different types of tailoring algorithms against each other.
  • Alternatives to Collaborative Filtering: The paper did not explore alternative approaches like using demographic and psychographic procedures to find "clones" for message recommendation.
  • Future Research: The authors suggest future research should explore more predictively powerful algorithms and investigate the effectiveness of micro-targeting. They also highlight the potential for long-term investment in algorithmic approaches to message recommendation due to the availability of large message archives and the ability to test messages across targeted populations.

Summary updated

Abstract

Recommendation algorithms are widely used in online cultural markets to provide personalized suggestions for products like books and movies. At the heart of the commercial success of recommendation algorithms is their ability to make an accurate prediction of a target person’s preferences for previously unseen items. Can these algorithms also be used to predict which health messages an individual will evaluate favorably, and thereby provide effective tailored communication to the person? Although there is evidence that message tailoring enhances persuasion, little research has examined the effectiveness of recommendation algorithms for tailored health interventions aimed at promoting behavior change. We developed a message tailoring algorithm to select smoking-related public service announcements (PSAs) for smokers, and experimentally test its effectiveness in predicting a target smoker’s evaluations of PSAs and encouraging smoking cessation. The tailoring algorithm was constructed using multiple levels of data on smokers’ PSA rating history, individual differences, content features of the PSAs, and other smokers’ PSA ratings. We conducted a longitudinal online experiment to examine its efficacy in comparison to two non-tailored methods: “best in show” (choosing messages most preferred by other smokers) and “off the shelf” (random selection from eligible ads). The results showed that the tailoring algorithm produced more accurate predictions of smokers’ message evaluations than the simple-average method used for the “best in show” approach. Smokers who viewed PSAs recommended by the tailoring algorithm were more likely than those receiving a random set to evaluate the PSAs favorably and quit smoking. There was no significant difference between the “best in show” and “off the shelf” methods in message assessment and quitting behavior.
BibTeX
@inproceedings{kim2019experimental,
  author = {hyun Suk Kim and Sijia Yang and Minji Kim and Brett Hemenway and Lyle Ungar and JOSEPH NICHOLAS CAPPELLA},
  title = {{An Experimental Study of Recommendation Algorithms for Tailored Health Communication}},
  booktitle = {Computational},
  year = {2019},
  doi = {10.5117/ccr2019.1.005.sukk},
}

Loading related content...

Figure