TL;DR
This paper tackles the challenge of classifying clickbait spoilers by exploring the use of webpage markup. While their DeBERTaV3 models didn't beat the baseline, they found that incorporating HTML markup, specifically <p> and <a> tags from Huffington Post articles, showed promise, though limitations in data and markup extraction hindered performance.
What Problem Does This Paper Address?
Clickbait articles are a pervasive annoyance on the internet, designed to lure users with misleading headlines and ultimately degrade the user experience and trust. SemEval-2023 Task 5 aimed to address this by classifying "spoiler types" (passage, phrase, or multi) within clickbait articles. The authors recognized that the way information is presented on a webpage, often influenced by design choices and advertising strategies, might hold clues to identifying these spoilers. They hypothesized that analyzing the webpage's underlying markup (HTML) could provide valuable signals for classification, going beyond just the text content.
Methods
The core of the authors' approach involved fine-tuning transformer-based language models, specifically BERTBASE and DeBERTaV3 (both BASE and LARGE variants). They experimented with different sequence lengths and combinations of text features like "postText" (the text accompanying the link on social media), "targetTitle" (the clickbait headline), and "targetParagraphs" (the main article content).
The novel aspect of their work was the integration of webpage markup. They crawled HTML from the original websites, focusing on extracting <p> (paragraph) and <a> (anchor) tags. This extracted markup was then used as an additional feature for their DeBERTaV3LARGE models. They also explored various dataset balancing strategies to address imbalances in the data, particularly the overrepresentation of Huffington Post articles and certain spoiler tag types.
Datasets
The primary dataset used was the Webis Clickbait 2022 Corpus, which contains 4000 clickbait articles split into training and validation sets. For their markup-based approach, they crawled markup specifically from Huffington Post articles within the training and validation datasets.
Key Insights and Ideas
- Markup Matters (Potentially): The authors' hypothesis that webpage markup could be a valuable signal for clickbait classification is a key insight. While their implementation had limitations, the idea of leveraging structural webpage elements is promising.
- Huffington Post Dominance is a Problem: The dataset's heavy reliance on Huffington Post articles creates a significant bias, impacting model generalization and potentially skewing results. This overrepresentation also means that specific spoiler tag distributions within that domain might not reflect the broader clickbait landscape.
- Data Quality is Crucial: The presence of embedded articles (e.g., from "Yahoo") and archived links within the dataset introduces noise and complicates markup analysis. Cleaning these out is essential for more robust models.
- Balancing Act: Addressing the imbalance in spoiler tag distribution (e.g., fewer "multi" spoilers) and domain representation is critical for building fair and effective classifiers.
Key Results
- The authors' best performing model, a DeBERTaV3LARGE trained on a combined dataset (including Huffington Post articles), achieved a balanced accuracy of 0.70, which was below the baseline accuracy of 0.74.
- Their markup-enriched approach, while promising in concept, resulted in the lowest accuracy (0.67), likely due to the limited scope of markup extracted (only
<p>and<a>tags) and the focus solely on Huffington Post articles. - Despite not surpassing the baseline, their team ranked among the top 10 for most measures and achieved a top 3 F1 score for classifying passage spoilers.
- They observed the highest recall of 0.79 for the full-combined dataset and the Huffington Post run in detecting multi-part spoilers.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Markup Extraction: They only extracted
<p>and<a>tags, which is a small fraction of a webpage's design elements. Future work should involve extracting more comprehensive markup. - Huffington Post Bias: The overrepresentation of Huffington Post articles and the limited scope of markup crawling from this single source significantly impacted their markup-based approach.
- Dataset Noise: Embedded articles and archived links introduce noise that hinders accurate analysis.
- Lack of Markup in Dataset: The original dataset did not include webpage markup, forcing them to crawl it retroactively, which is a labor-intensive and potentially incomplete process.
Future work should focus on:
- Re-annotating the dataset: Including full webpage markup would allow for better utilization of models like MarkupLM and TIE.
- Comprehensive markup extraction: Capturing a wider range of HTML elements.
- Dataset balancing: Addressing domain and tag imbalances more effectively.
- Removing noisy data: Cleaning the dataset of embedded and archived articles.
Summary updated