AI-based Approach for Safety Signals Detection from Social Networks: Application to the Levothyrox Scandal in 2017 on Doctissimo Forum

Valentin Roche, Jean-Philippe Robert, Hanan SalamarXiv 2022 · DOI · arXiv

Introduction

Pharmacovigilance—the science of monitoring drug safety—has traditionally relied on spontaneous reports from healthcare professionals. However, this system often misses the patient's perspective due to the "white coat effect," a phenomenon where patients feel hesitant to share subtle or subjective side effects with clinicians. Consequently, many patients turn to anonymous online medical forums to share experiences and seek advice.

This paper investigates the 2017 Levothyrox scandal in France, where a change in the excipients of a common thyroid medication triggered a massive wave of adverse drug reactions (ADRs). By analyzing data from the Doctissimo forum, the researchers explore whether artificial intelligence can detect such safety signals earlier than traditional media and regulatory bodies.

Methodology

The researchers developed a hybrid approach combining traditional Natural Language Processing (NLP) with a novel visual classification technique.

1. NLP-Based Safety Indicators

The study first employed standard NLP techniques to monitor linguistic shifts, including:

  • Frequency Analysis: Tracking the occurrence of specific n-grams and keywords.
  • Semantic Similarity: Utilizing FastText word embeddings to measure the "distance" between terms. A key observation was the convergence of terms like "New" and "Side Effect" during the crisis.
  • Sentiment Analysis: Implementing a classifier trained on French-translated sentiment data to monitor the emotional valence of forum posts over time.

2. The Word Cloud CNN (WC-CNN)

The most innovative contribution of this work is the WC-CNN model. Rather than processing raw text directly through a sequence model, the authors converted forum posts into word cloud images. These images represent word frequency and importance through visual size. These word clouds were then used as input for a Convolutional Neural Network (CNN)—a model architecture typically reserved for image recognition tasks. The CNN was trained to classify these visual representations of time periods (daily, weekly, or monthly) as either "Normal" or "Abnormal."

Data and Materials

The primary dataset consisted of 110,260 comments extracted from the "Thyroid and Endocrine Problems" section of the Doctissimo forum, covering a twenty-year span from 2000 to 2020. To address the lack of medical-specific sentiment datasets in French, the authors utilized a translated dataset of labeled tweets to train their sentiment analysis component.

Results and Key Insights

The study yielded several critical insights into the dynamics of the Levothyrox scandal:

  • Detection Timeline: The WC-CNN model identified "abnormal" activity as early as May 2017. This predates the peak of the public outcry and mainstream media coverage, which occurred in August 2017, by approximately three months.
  • Model Performance: The researchers found that temporal resolution significantly impacted accuracy. Monthly word clouds provided the most stable signal, achieving a classification accuracy of 75%. Daily and weekly data proved too sparse and noisy for effective classification.
  • Linguistic Shifts: During the height of the scandal, the frequency of the term "Old Formula" (ancien formule) increased 160-fold compared to pre-scandal levels. Furthermore, regex-based extraction showed that mentions of side effects rose from a baseline of 6 per day to 25 per day during the crisis.
  • Semantic Convergence: In "normal" periods, words like "New" and "Side Effect" are linguistically distant. During the scandal, these terms became inseparable in the vector space, indicating a strong thematic link in patient discourse.

Discussion and Limitations

The results demonstrate that social media analysis can serve as a powerful "early warning system" for pharmacovigilance. By converting unstructured text into visual data, the WC-CNN effectively filters out the "noise" inherent in forum discussions.

However, several limitations remain. The current model relies on a manually curated dictionary of irrelevant words, which limits its scalability. Future work should integrate standardized medical ontologies like the ICD-10 or ATC classifications. Additionally, the study's focus on a single forum and a single drug scandal suggests that further validation across different platforms (e.g., Twitter, Reddit) and various medication classes is necessary to ensure the robustness of the approach. Finally, researchers must account for "herd behavior," where media coverage itself can drive an increase in online reporting, potentially confounding the initial safety signal.

Conclusion

This research highlights the potential of deep learning to transform patient-generated data into actionable safety signals. By identifying the Levothyrox crisis months before it reached its peak, the WC-CNN approach offers a promising path toward more responsive and patient-centered drug safety monitoring.

Summary updated

Abstract

Social media can be an important source of information facilitating the detection of new safety signals in pharmacovigilance. Various approaches have investigated the analysis of social media data using AI such as NLP techniques for detecting adverse drug events. Existing approaches have focused on the extraction and identification of Adverse Drug Reactions, Drug-Drug Interactions and drug misuse. However, non of the works tackled the detection of potential safety signals by taking into account the evolution in time of relevant indicators. Moreover, despite the success of deep learning in various healthcare applications, it was not explored for this task. We propose an AI-based approach for the detection of potential pharmaceutical safety signals from patients' reviews that can be used as part of the pharmacovigilance surveillance process to flag the necessity of an in-depth pharmacovigilance investigation. We focus on the Levothyrox case in France which triggered huge attention from the media following the change of the medication formula, leading to an increase in the frequency of adverse drug reactions normally reported by patients. Our approach is two-fold. (1) We investigate various NLP-based indicators extracted from patients' reviews including words and n-grams frequency, semantic similarity, Adverse Drug Reactions mentions, and sentiment analysis. (2) We propose a deep learning architecture, named Word Cloud Convolutional Neural Network (WC-CNN) which trains a CNN on word clouds extracted from the patients comments. We study the effect of different time resolutions and different NLP pre-processing techniques on the model performance. Our results show that the proposed indicators could be used in the future to effectively detect new safety signals. The WC-CNN model trained on word clouds extracted at monthly resolution outperforms the others with an accuracy of 75%.
BibTeX
@misc{roche2022ai,
  author = {Valentin Roche and Jean-Philippe Robert and Hanan Salam},
  title = {{AI-based Approach for Safety Signals Detection from Social Networks: Application to the Levothyrox Scandal in 2017 on Doctissimo Forum}},
  year = {2022},
  eprint = {2203.03538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.03538},
}
Figure