TL;DR
This paper introduces a novel neuro-symbolic approach for classifying propaganda techniques in memes, leveraging a knowledge distillation framework that combines a transformer-based text encoder and a CNN-based image encoder with a logic rule layer. The goal is to improve classification consistency by enforcing hierarchical relationships between propaganda techniques, leading to fewer hierarchy violations with a slight boost in overall performance.
What Problem Does This Paper Address?
Propaganda is a significant tool in modern information campaigns, and memes have become a prevalent medium for its dissemination. Effectively identifying and curtailing propaganda requires systems that can accurately detect the persuasive techniques employed within these memes. SemEval-2024 Task 4 specifically targets this challenge by asking participants to classify persuasion techniques in memes, considering both textual and visual content. A key complexity is the hierarchical nature of these techniques, where some are sub-categories of others. This hierarchy adds a layer of difficulty to classification, as predictions must not only be accurate but also consistent with the established relationships within the taxonomy. Existing methods often treat each technique independently, potentially leading to inconsistent predictions that violate these hierarchical structures.
Methods
The proposed approach employs a student-teacher knowledge distillation framework. The "student" is a neural network that combines a DeBERTa model for text and a ResNet-50 for image processing. These modalities are concatenated and fed into a final feed-forward network for classification. The "teacher" component is a logic rule layer that encodes the hierarchical relationships between propaganda techniques. This layer uses softened logic rules to represent these constraints. The knowledge from the teacher is distilled into the student network through a joint training process, aiming to improve the student's ability to respect the hierarchy. This is achieved by minimizing a combined loss function that includes both the standard classification loss (comparing student predictions to gold labels) and a loss that encourages the student's predictions to align with the teacher's logic-rule-based outputs. The logic rules are designed to penalize predictions where a technique is identified but its parent technique is not, thus enforcing hierarchical consistency.
Datasets
The research utilizes datasets provided by the SemEval-2024 Task 4 organizers. These datasets include labeled examples for training, validation, and testing. The task involves three sub-tasks:
- Sub-task 1: Identifying persuasion techniques in textual content.
- Sub-task 2a: Identifying persuasion techniques in both textual and visual content.
- Sub-task 2b: Identifying whether any persuasion technique is present in visual content.
The paper focuses on the English versions of sub-tasks 1 and 2a. The training data consists of 7,000 labeled examples, with 500 for validation, 1,000 for development, and 1,500 for testing.
Key Insights and Ideas
- Hierarchy Matters: Explicitly encoding the hierarchical structure of propaganda techniques into the model significantly reduces prediction inconsistencies (hierarchy violations).
- Logic Distillation Trade-off: While logic rules improve consistency, overly strict enforcement can sometimes negatively impact the performance on specific, fine-grained "leaf" nodes of the hierarchy.
- Neuro-Symbolic Synergy: Combining neural networks with symbolic logic rules offers a promising direction for tasks with inherent structural constraints, potentially leading to more robust and interpretable models.
- Modality Contribution: For sub-task 2a (text + image), the image modality appears to be more beneficial for classifying higher-level hierarchical categories (like Ethos, Pathos) than for specific leaf-level techniques, and can sometimes even confuse the model for certain leaf techniques.
Key Results
- Sub-task 1 (English Test Set): The proposed "Pauk" system ranked 15th out of 33 submissions, achieving a hierarchical F1 score.
- Sub-task 2a (English Test Set): The "Pauk" system ranked 7th out of 14 submissions.
- Hierarchy Violations: The neuro-symbolic model consistently demonstrated fewer hierarchical violations compared to the baseline model across validation sets for both sub-tasks, with stronger rule confidences leading to fewer violations.
- F1 Score Impact: While logic rules improved consistency, they resulted in a slight improvement or a minor decrease in micro F1 scores compared to the baseline, depending on the rule confidence. A rule confidence of 100 seemed to offer the best compromise.
- Leaf Node Performance: When evaluating only on leaf nodes, the neuro-symbolic model showed lower F1 scores, suggesting that the consistency enforced by the logic rules might not directly translate to better performance on the most specific categories.
Limitations and Future Work
The authors acknowledge that the logic rule layer, while improving consistency, did not necessarily lead to better predictions on the specific leaf nodes of the hierarchy. They also note that the KL divergence loss function can sometimes lead the network to lower predictions to minimize loss, rather than raising ancestor probabilities as intended.
Future work suggestions include:
- Exploring alternative logic rule integrations and loss function formulations to better ensure the desired learning behavior.
- Investigating other image processing methods beyond ResNet and more advanced mechanisms for combining textual and visual features (e.g., attention).
- Developing additional logic rules that specifically target textual content indicative of certain persuasion techniques, especially for low-coverage techniques.
Summary updated