TL;DR
This paper presents systems for SemEval-2023 Task 5, tackling clickbait classification and question answering. They leverage pre-trained RoBERTa models, enhanced with Named Entity Recognition (NER) and custom features for classification, and a RoBERTa-SQuAD2.0 model with rule-based logic for spoiler generation. While not outperforming all baselines, their approach offers valuable insights into combining transformer models with domain-specific features for clickbait analysis.
What Problem Does This Paper Address?
Clickbait is designed to pique user curiosity, often leading to a click to reveal the actual content. SemEval-2023 Task 5, "Clickbait Spoiling," aims to address this by:
- Classifying clickbait posts: Categorizing them into types that can be answered with a short phrase, a longer passage, or a list/enumeration.
- Generating spoiler answers: Providing a concise answer to the clickbait's implied question, essentially "spoiling" it to save users a click.
This task is crucial for information retrieval, helping users get quick answers without engaging with potentially misleading content. The challenge lies in accurately identifying spoiler types and generating relevant, concise answers from linked content.
Methods
The authors developed two main systems, one for each sub-task, primarily using pre-trained transformer models:
- Sub-task 1: Spoiler Type Classification:
- Core Model: RoBERTa (a robustly optimized BERT pretraining approach).
- Enhancements: * Named Entity Recognition (NER): Using spaCy to identify and categorize entities (organizations, persons, dates, locations) to enrich the input.
- Spoiler-Title Ratio: A metric comparing the length of the post's title to its target description, hypothesizing that shorter ratios indicate passage spoilers and longer ones indicate phrase spoilers.
- Enumeration Check: A regex-based check for lists or enumerations within the text, intended to help identify "multi" type spoilers.
- Input Reformulation: Experimented with feeding the model natural sentences enriched with these features, both as individual entities and as simple counts, to manage input length.
- Sub-task 2: Spoiler Generation:
- Core Model: RoBERTa-SQuAD2.0, a model fine-tuned for extractive question answering.
- Enhancements: * Rule-Based Approach: Combined with the transformer model, this approach specifically targets "multi" type spoilers. It uses enumeration checks and regex to identify and extract relevant list items from the text.
- Two Versions (v1 & v2): v1 used only the RoBERTa-SQuAD2.0 model, while v2 incorporated the rule-based logic for multi-type spoilers.
Datasets
- Webis Clickbait Spoiling Corpus 2022: This is the primary dataset used for both training and evaluation. It contains 5000 clickbait posts from social media (Facebook, Reddit, Twitter) with annotations for spoiler types (sub-task 1) and manually created spoiler answers (sub-task 2).
- SQuAD2.0: Used to fine-tune the RoBERTa model for extractive question answering in sub-task 2.
The authors also mention using a split of the validation set for an "improvised test data set" for sub-task 1 due to the test set being undisclosed until after the task.
Key Insights and Ideas
- NER's Mixed Impact: While NER was intended to enrich the data, its inclusion in sub-task 1 did not lead to improved performance compared to a simpler transformer baseline. This suggests that simply adding named entities might not always benefit classification, especially if information is lost during input reformulation.
- Spoiler-Title Ratio as a Heuristic: The hypothesis that spoiler-title ratio can distinguish between phrase and passage spoilers is interesting. However, the paper notes a potential issue with precision in this metric, which might have led to overfitting.
- Rule-Based Logic for Multi-Type Spoilers: The most significant insight is the positive impact of the rule-based approach for handling "multi" type spoilers in sub-task 2. This rule-based logic, when combined with the transformer model, led to a noticeable performance increase, particularly for multi-type spoiler generation.
- Transformer Baselines are Strong: The paper consistently highlights that while their enhanced systems show promise, they often struggled to outperform the provided "naive" and "transformer" baselines, underscoring the power of well-tuned pre-trained models.
Key Results
- Sub-task 1 (Classification): Achieved a balanced accuracy of 0.593 on the test dataset. The system performed best on "passage" spoilers (F1 score of 0.70) but struggled with "phrase" spoilers (F1 score of 0.43) due to low recall.
- Sub-task 2 (Generation): Achieved a BLEU score of 0.322 on the test dataset. System v2 (with rule-based logic) slightly outperformed v1, showing a 1% increase in BLEU score and a more significant improvement in generating multi-type spoiler answers.
Limitations and Future Work
- Performance Gap: The authors acknowledge that their systems did not outperform the challenge organizers' basic transformer baselines in either sub-task.
- NER Feature Loss: The decision to represent named entities as counts instead of individual names for input reformulation might have led to information loss, negatively impacting sub-task 1 performance.
- Spoiler-Title Ratio Precision: The paper suggests that the spoiler-title ratio metric might have been too precise, potentially leading to overfitting.
- Future Work:
- Further research into rule-based logic, incorporating semantic or syntactic patterns, for improving transformer-based approaches.
- Developing a unified system that combines spoiler type classification with rule-based question answering, where the predicted spoiler type dictates the question-answering strategy.
- Exploring alternative ways to combine features from both sub-tasks.
Summary updated