Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media

Thales Bertaglia, Stefan Huber, Catalina Goanta, Gerasimos Spanakis, Adriana IamnitchiCommunications 2023 · DOI · OpenAlex

Abstract

Regulatory bodies worldwide are intensifying their efforts to ensure transparency in influencer marketing on social media through instruments like the Unfair Commercial Practices Directive (UCPD) in the European Union, or Section 5 of the Federal Trade Commission Act. Yet enforcing these obligations has proven to be highly problematic due to the sheer scale of the influencer market. The task of automatically detecting sponsored content aims to enable the monitoring and enforcement of such regulations at scale. Current research in this field primarily frames this problem as a machine learning task, focusing on developing models that achieve high classification performance in detecting ads. These machine learning tasks rely on human data annotation to provide ground truth information. However, agreement between annotators is often low, leading to inconsistent labels that hinder the reliability of models. To improve annotation accuracy and, thus, the detection of sponsored content, we propose using chatGPT to augment the annotation process with phrases identified as relevant features and brief explanations. Our experiments show that this approach consistently improves inter-annotator agreement and annotation accuracy. Additionally, our survey of user experience in the annotation task indicates that the explanations improve the annotators' confidence and streamline the process. Our proposed methods can ultimately lead to more transparency and alignment with regulatory requirements in sponsored content detection.

Introduction

Influencer marketing has become a dominant force in digital advertising, yet it remains a "Wild West" regarding legal disclosures. While regulations like the EU’s Unfair Commercial Practices Directive require influencers to clearly label ads, the sheer volume of content makes manual enforcement impossible. To automate this, machine learning models are necessary. However, these models are only as good as the data they are trained on. Currently, human annotators often disagree on whether a post is "sponsored" or just a "personal recommendation," leading to low inter-annotator agreement and "noisy" datasets.

This research addresses the "ground truth" problem by developing a "human-in-the-loop" framework that uses ChatGPT to help people better identify sponsored social media posts. By providing annotators with AI-generated explanations and highlighted "key indicators," the study aims to boost both the accuracy and consistency of human labels—a crucial step for training more reliable models to enforce advertising transparency regulations.

Methodology

The proposed annotation framework is augmented by Large Language Models (LLMs) and involves several stages:

Model Selection and Prompting

The researchers compared three classifiers—Logistic Regression (TF-IDF), a fine-tuned BERT model, and GPT-3.5—to determine the most effective baseline for detecting undisclosed ads. To generate helpful explanations, they utilized Chain-of-Thought (CoT) prompting. Instead of a simple classification request, the model was instructed to first identify relevant words (e.g., brand names, discount codes), generate a brief explanation of the partnership, and finally provide a classification.

User Study and Datasets

A custom-curated Instagram Dataset of 294,600 posts from 100 US-based influencers was collected via CrowdTangle. From this, an evaluation set of 1,283 posts was manually labeled to benchmark initial classifiers. For the user study, 11 annotators with varying expertise (novices, intermediates, and legal experts) labeled a subset of 200 posts. Some participants labeled posts with only the caption, while others had access to the AI-generated explanations.

Results

The initial model benchmarking revealed a stark difference in performance across architectures. GPT-3.5 significantly outperformed traditional and transformer-based baselines in identifying undisclosed advertisements.

Comparison of model performance metrics for undisclosed ad detection.

As shown in the table above, GPT-3.5 achieved an accuracy of 88.98%, vastly outperforming BERT (10.85%) and Logistic Regression (28.71%).

Impact on Human Annotation

The integration of AI explanations had a measurable impact on human performance:

  • Inter-Annotator Agreement: Using AI explanations led to a 15.65% increase in Krippendorff’s Alpha ($\alpha$) and a 17.20% increase in absolute agreement.
  • Labeling Accuracy: Accuracy in identifying posts with clear disclosures (attention checks) improved by 3.45%, suggesting AI helps humans maintain focus.
  • The "Expert" Boost: Legal experts saw the largest consistency improvement (18.86% increase in $\alpha$), indicating that even domain experts benefit from AI "pointing" at specific evidence.

User Experience Insights

Annotators reported that "Key Indicators" (specific tokens like @brand or "discount code") were more helpful than long-form reasoning. Furthermore, 87.5% of participants felt more confident in their decisions with AI assistance, reducing the cognitive load of the subjective task.

Discussion and Limitations

While the results are promising, the study highlights several areas for future research:

  • Automation Bias: Annotators showed a shift toward agreeing with GPT predictions when labeling the same posts twice, raising questions about whether the AI is educating the human or simply swaying them.
  • Technical Constraints: The framework currently relies on closed-source models (GPT-3.5) and text-only analysis. Future work should explore open-source alternatives like LLaMA and multimodal approaches that incorporate image analysis, as Instagram is primarily a visual platform.
  • Sample Size: The small number of annotators, particularly in the legal expert subgroup, suggests the need for larger-scale validation.

Ultimately, this "human-in-the-loop" approach demonstrates that AI can serve as a powerful tool for improving data quality in regulatory contexts, leading to more transparent and compliant social media environments.

Summary updated

BibTeX
@inproceedings{bertaglia2023closing,
  author = {Thales Bertaglia and Stefan Huber and Catalina Goanta and Gerasimos Spanakis and Adriana Iamnitchi},
  title = {{Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media}},
  booktitle = {Communications},
  year = {2023},
  doi = {10.1007/978-3-031-44067-0\_11},
}
Figure