LLM-Based Semantic Augmentation for Harmful Content Detection

Elyas Meguellati, Assaad Oussama Zeghina, Shazia Sadiq, Gianluca Demartini

LLM-based Semantic Augmentation for Harmful Content Detection

Abstract

Detecting harmful content like propaganda and hate speech on social media is challenging due to the high reliance on subtle context. This paper proposes a semantic augmentation framework that uses Large Language Models (LLMs) to "clean" noisy image captions and "explain" persuasive techniques. By integrating these LLM-generated features into a supervised learning pipeline, the authors achieve significant performance gains, outperforming both zero-shot LLM baselines and traditional multimodal models.

Introduction

The detection of harmful online content—including propaganda, toxicity, and hateful memes—remains a critical challenge. The difficulty stems from the fact that harm is often encoded in nuanced rhetorical strategies rather than explicit keywords. While current research often uses LLMs to generate synthetic training data, this study explores Semantic Augmentation: using LLMs to improve the quality and depth of the feature space by providing context-rich explanations and refining noisy automated inputs.

Methodology

The proposed framework integrates LLMs into the machine learning pipeline through three primary mechanisms:

  1. LLM-based Cleaning: Automated image captioning tools (e.g., BLIP) often produce noisy or irrelevant descriptions. The framework uses LLMs to refine these captions, correcting errors and filtering out "hallucinated" content.
  2. Semantic Augmentation (Explanations): LLMs are prompted to analyze content and generate concise explanations of the underlying rhetorical or psychological strategies. This latent context is then appended to the input for a downstream supervised model (e.g., a BART decoder).
  3. Trigger-based Augmentation: To navigate LLM safety filters that might block detailed analysis of offensive content, a "trigger" strategy extracts specific themes and keywords while preserving the original text's features for the downstream classifier.
Overview of the six experimental conditions, ranging from zero-shot LLMs to the proposed LLM-augmented pipeline (Condition 6).

The researchers evaluated six distinct conditions to isolate the impact of each component, as illustrated in the pipeline overview above.

Experimental Setup

The framework was tested across three benchmarks:

  • SemEval 2024 Task 4 (Persuasive Memes): Identifying 22 persuasion techniques in memes.
  • Google Jigsaw Toxic Comments: A text-only dataset for toxicity detection.
  • Facebook Hateful Meme Challenge: A multimodal dataset for binary hate speech classification.

Key Results

The experimental results demonstrate that semantic augmentation significantly enhances model performance across various tasks.

Comparison of H-F1 scores on the SemEval Persuasive Memes task, showing the "Best Config" (LLM + decoder) outperforming baselines and zero-shot models.
  • Performance Gains: On the SemEval task, the "LLM + decoder" configuration reached a 62.2% H-F1 score, a substantial improvement over the zero-shot GPT-4o (54.2%) and human-annotated baselines (57.8%).
  • Cost Efficiency: LLM-generated annotations were significantly more affordable than human crowdsourcing, costing between $3 and $30 for the full dataset compared to over $1,100 for a human-annotated subset.
  • Generalizability: The trigger-based approach improved F1 scores in toxicity detection (25.3) compared to specialized models like DistilBERT (21.3).

Discussion and Insights

The study highlights that context is more valuable than quantity. Providing a high-quality explanation of a meme's intent is often more effective for a supervised model than simply increasing the volume of raw training data. Furthermore, while top-tier models like GPT-4o may struggle with direct classification of complex propaganda, they excel at describing these techniques, serving as powerful feature extractors for specialized classifiers.

Limitations

The authors note that the current pipeline is text-centric, converting visual data into captions early in the process. Future research could explore tighter multimodal integration. Additionally, the reliance on LLM explanations increases input sequence length, impacting computational overhead during training and inference.

Conclusion

By shifting the role of LLMs from data generators to semantic refiners, this research provides a robust and cost-effective path for improving harmful content detection. The combination of cleaned captions and LLM-generated explanations allows smaller supervised models to capture the nuanced meanings essential for identifying sophisticated online harm.

Summary updated

BibTeX
@inproceedings{meguellati2025llm,
  author = {Elyas Meguellati and Assaad Oussama Zeghina and Shazia Sadiq and Gianluca Demartini},
  title = {{LLM-Based Semantic Augmentation for Harmful Content Detection}},
  booktitle = {ICWSM},
  year = {2025},
  url = {https://openreview.net/forum?id=ssgDCWpwTE},
}
Figure