Can Social Media Inform Dietary Approaches for Health Management? A Dataset and Benchmark for Low-Carb Diet

Skyler Zou, Xiang Dai, Grant Brinkworth, Pennie Taylor, Sarvnaz KarimiACL 2023

TL;DR

This paper introduces a novel annotated dataset of Reddit posts about low-carb diets, designed to help health professionals understand user experiences, challenges, and advice. They benchmarked several NLP models, finding that while large language models like ChatGPT show promise, supervised models still outperform them for this specific task.

What Problem Does This Paper Address?

While low-carb diets are gaining traction for their health benefits, particularly for conditions like Type II Diabetes, there's a significant gap in understanding the real-world experiences of individuals following these diets. Existing research often relies on controlled clinical trials, which don't capture the nuances of daily challenges, perceived health effects, and the social support networks that emerge online. Health professionals could benefit from insights gleaned from social media to better educate and support patients. However, a lack of structured, annotated data from these platforms makes it difficult to apply NLP techniques to extract this valuable information.

Methods

The researchers developed an annotation schema with three key categories: Health Response (physiological or psychological effects), Barrier (challenges to diet progression), and Advice (recommendations or tips). They manually annotated posts from the r/lowcarb subreddit using Label Studio. To benchmark the dataset's utility, they evaluated three types of classifiers:

  • NB-SVM: A statistical Support Vector Machine approach using Naive Bayes log-count ratios as features.
  • RoBERTa: A transformer-based encoder model, fine-tuned for multi-label text classification. Both base and large versions were tested.
  • ChatGPT: An off-the-shelf large language model, used in both zero-shot and few-shot (5-shot) settings by constructing prompts.

They employed both "hard" evaluation (majority vote for gold labels) and "soft" evaluation (comparing predictions against individual annotator labels) to assess classifier performance.

Datasets

The primary dataset introduced in this paper is a collection of 1570 unique comments and 1210 unique submissions from the r/lowcarb subreddit. This dataset was manually annotated by five annotators (two dietary experts, three computer scientists) into four categories: Health Response, Barrier, Advice, and Other. The annotated data, along with scripts for downloading Reddit posts, is publicly available.

Key Insights and Ideas

  • Social media as a rich source of real-world dietary insights: The r/lowcarb subreddit provides a valuable platform for understanding the practicalities and personal experiences of low-carb dieting, complementing clinical trial data.
  • The importance of a well-defined annotation schema: Creating categories relevant to healthcare professionals (health responses, barriers, advice) is crucial for extracting actionable information.
  • Supervised models outperform LLMs for this task: Despite the hype around large language models, fine-tuned supervised models (RoBERTa) demonstrated significantly better performance on this specific dataset.
  • Hard evaluation can overestimate classifier performance: Using a simple majority rule for gold labels can mask errors, as models predicting "Other" might be penalized less than they should be in a more nuanced evaluation.
  • ChatGPT's instability: The zero-shot and few-shot performance of ChatGPT was found to be sensitive to prompt construction and randomness, with lower temperature settings improving stability.

Key Results

  • RoBERTa-Large achieved the highest F1 scores in both hard (77.4 Micro F1) and soft (64.4 Micro F1) evaluations, outperforming NB-SVM and ChatGPT.
  • ChatGPT (0-shot) performed poorly, with Micro F1 scores of 55.3 (hard) and 51.2 (soft), suggesting that its general knowledge wasn't sufficient without specific fine-tuning or strong in-context examples.
  • 5-shot learning did not significantly improve ChatGPT's performance over zero-shot, except for the "Advice" category, indicating it relied more on its pre-training than the provided examples.
  • Inter-annotator agreement was moderate for "Health Response" (kappa 0.59) and fair for "Barrier" (kappa 0.37), with dietary experts showing more disagreement, possibly due to their deeper domain knowledge leading to different interpretations.

Limitations and Future Work

  • Selection Bias: The r/lowcarb subreddit may have a selection bias towards users with more positive experiences.
  • API Discontinuation: The Pushshift API used for data collection has since been discontinued, meaning future data collection might require different methods.
  • ChatGPT Instability: The paper highlights the need for controlling randomness in LLM outputs, suggesting lower temperature settings and careful prompt engineering.
  • Future Work: The authors suggest investigating other data sources and exploring more advanced NLP techniques to further leverage social media data for health management insights. They also note the potential for using this dataset to inform dietitians in clinical practice.

Summary updated

BibTeX
@inproceedings{zou2023can,
  author = {Skyler Zou and Xiang Dai and Grant Brinkworth and Pennie Taylor and Sarvnaz Karimi},
  title = {{Can Social Media Inform Dietary Approaches for Health Management? A Dataset and Benchmark for Low-Carb Diet}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure