TL;DR
This paper tackles the challenge of automatically identifying the type of spoiler needed for clickbait headlines. The authors fine-tuned large pre-trained language models (RoBERTa and DeBERTa) on post and document features, achieving top-tier results in the SemEval-2023 Clickbait Spoiling task, particularly excelling in passage spoiler classification.
What Problem Does This Paper Address?
Clickbait headlines are designed to lure users into clicking by exaggerating or distorting content. While this drives traffic, the actual content is often trivial, leading to user dissatisfaction. The SemEval-2023 Clickbait Spoiling task aims to combat this by providing "spoilers" – short previews that reveal the core content without requiring a click. This task is divided into two parts: classifying the type of spoiler (phrase, passage, or multipart) and generating the spoiler text. This paper focuses on the first subtask: spoiler type classification, addressing the need for accurate and efficient identification of spoiler categories.
Methods
The core of the UMUTeam's approach for spoiler type classification relies on transfer learning, specifically fine-tuning pre-trained Transformer-based language models. They experimented with two powerful models:
- RoBERTa-large: A large-scale variant of RoBERTa with 355 million parameters, known for its ability to capture complex text patterns.
- DeBERTa-v2-xlarge: A significantly larger model (3.3 billion parameters) developed by Microsoft Research, building upon BERT with disentangled attention.
The system preprocesses clickbait data by cleaning social media language, replacing hashtags/mentions, and removing email links. Crucially, they extract two types of features:
- Post-based features: Combining the clickbait post text and its target title.
- Document-based features: Extracting relevant parts from the linked document. To handle potentially long documents, they used a Sentence-Transformer model to identify the most relevant paragraph (under 200 tokens) to the post text.
They then fine-tuned RoBERTa-large and DeBERTa-v2-xlarge on these feature sets separately. Finally, they explored ensemble learning by combining the predictions from models trained on post-based and document-based features, using both mean and max strategies to see if combining features improved performance.
Datasets
The primary dataset used was provided by the organizers for SemEval-2023 Task 5. It consists of 5000 clickbait posts, split into training (3200), validation (800), and testing (1000) sets. The dataset includes the clickbait post text, target title, and the manually cleaned version of the linked document. A significant imbalance was noted, particularly with the "multipart" spoiler type being less frequent than "phrase" or "passage" types in the training set.
Key Insights and Ideas
- Post features dominate: The most striking insight is that features derived solely from the clickbait post (title and text) significantly outperform features extracted from the linked document for spoiler type classification. This suggests that the inherent characteristics of the clickbait itself are more indicative of the spoiler type than the actual content of the linked page.
- Document content is less informative: The paper's findings indicate that the linked documents, despite containing the actual content, do not effectively complement the post text for this specific classification task. This might be due to the nature of clickbait, where the title is designed to be misleading, and the document content might not directly correlate with the type of spoiler needed.
- Larger models matter: DeBERTa-v2-xlarge, being a much larger model, generally performs better than RoBERTa-large, especially when trained on post-based features.
- Ensemble learning limitations: While ensemble learning was explored, it did not yield improvements over single-feature models, reinforcing the idea that post-based features are the most critical.
Key Results
The UMUTeam achieved excellent results in Task 1, placing within the Top 10 for most metrics and reaching Top 3 in F1 score for passage spoiler ranking.
- Post-based features: Models trained on post-based features significantly outperformed those trained on document-based features. For instance, DeBERTa-v2 achieved a 21.01% higher performance on post-based features compared to document-based features.
- Best performing model: The fine-tuned DeBERTa-v2 model using post-based features was the best performing model for Task 1, achieving a weighted F1-score of 0.7259 on the validation set.
- Post-evaluation with combined features: In a post-evaluation phase, combining post and document features did show improvement over single-feature models and ensemble learning. The DeBERTa-large model achieved a weighted F1-score of 0.74153 in this scenario, outperforming the best fine-tuning model by 1.563%. This suggests that while document features alone are weak, they can contribute when combined intelligently with post features.
- Confusion Matrix: The confusion matrix revealed that the best model misclassified 7.25% of passage spoilers as multipart and 8.75% of multipart spoilers as passage spoilers, indicating a potential area for improvement.
Limitations and Future Work
- Dataset Imbalance: The inherent imbalance in the dataset, particularly the scarcity of "multipart" spoilers, is a limitation that could affect model performance.
- Confusing Spoiler Types: The model struggles to differentiate between "passage" and "multipart" spoilers, as highlighted by the confusion matrix.
- Future Work:
- Spanish and Multilingual Variants: The authors plan to apply Spanish and multilingual variants of RoBERTa and DeBERTa.
- Domain-Specific Clickbait: They intend to extend the dataset with clickbait from the financial domain in Spanish and explore clickbait in satirical media.
- Ideology Impact: They aim to measure the impact of clickbait on datasets concerning political ideology.
- Feature Engineering: To address the confusion between passage and multipart spoilers, they plan to incorporate volumetric and lexical diversity features.
Summary updated