LLM-based Semantic Augmentation for Harmful Content Detection
Abstract
Detecting harmful content like propaganda and hate speech on social media is challenging due to the high reliance on subtle context. This paper proposes a semantic augmentation framework that uses Large Language Models (LLMs) to "clean" noisy image captions and "explain" persuasive techniques. By integrating these LLM-generated features into a supervised learning pipeline, the authors achieve significant performance gains, outperforming both zero-shot LLM baselines and traditional multimodal models.
Introduction
The detection of harmful online content—including propaganda, toxicity, and hateful memes—remains a critical challenge. The difficulty stems from the fact that harm is often encoded in nuanced rhetorical strategies rather than explicit keywords. While current research often uses LLMs to generate synthetic training data, this study explores Semantic Augmentation: using LLMs to improve the quality and depth of the feature space by providing context-rich explanations and refining noisy automated inputs.
Methodology
The proposed framework integrates LLMs into the machine learning pipeline through three primary mechanisms:
- LLM-based Cleaning: Automated image captioning tools (e.g., BLIP) often produce noisy or irrelevant descriptions. The framework uses LLMs to refine these captions, correcting errors and filtering out "hallucinated" content.
- Semantic Augmentation (Explanations): LLMs are prompted to analyze content and generate concise explanations of the underlying rhetorical or psychological strategies. This latent context is then appended to the input for a downstream supervised model (e.g., a BART decoder).
- Trigger-based Augmentation: To navigate LLM safety filters that might block detailed analysis of offensive content, a "trigger" strategy extracts specific themes and keywords while preserving the original text's features for the downstream classifier.
The researchers evaluated six distinct conditions to isolate the impact of each component, as illustrated in the pipeline overview above.
Experimental Setup
The framework was tested across three benchmarks:
- SemEval 2024 Task 4 (Persuasive Memes): Identifying 22 persuasion techniques in memes.
- Google Jigsaw Toxic Comments: A text-only dataset for toxicity detection.
- Facebook Hateful Meme Challenge: A multimodal dataset for binary hate speech classification.
Key Results
The experimental results demonstrate that semantic augmentation significantly enhances model performance across various tasks.
- Performance Gains: On the SemEval task, the "LLM + decoder" configuration reached a 62.2% H-F1 score, a substantial improvement over the zero-shot GPT-4o (54.2%) and human-annotated baselines (57.8%).
- Cost Efficiency: LLM-generated annotations were significantly more affordable than human crowdsourcing, costing between $3 and $30 for the full dataset compared to over $1,100 for a human-annotated subset.
- Generalizability: The trigger-based approach improved F1 scores in toxicity detection (25.3) compared to specialized models like DistilBERT (21.3).
Discussion and Insights
The study highlights that context is more valuable than quantity. Providing a high-quality explanation of a meme's intent is often more effective for a supervised model than simply increasing the volume of raw training data. Furthermore, while top-tier models like GPT-4o may struggle with direct classification of complex propaganda, they excel at describing these techniques, serving as powerful feature extractors for specialized classifiers.
Limitations
The authors note that the current pipeline is text-centric, converting visual data into captions early in the process. Future research could explore tighter multimodal integration. Additionally, the reliance on LLM explanations increases input sequence length, impacting computational overhead during training and inference.
Conclusion
By shifting the role of LLMs from data generators to semantic refiners, this research provides a robust and cost-effective path for improving harmful content detection. The combination of cleaned captions and LLM-generated explanations allows smaller supervised models to capture the nuanced meanings essential for identifying sophisticated online harm.
Summary updated