Brooke-English at SemEval-2023 Task 5: Clickbait Spoiling

Shirui TangACL 2023

TL;DR

This paper tackles the challenge of "clickbait spoiling" – generating a short, satisfying text that answers the curiosity piqued by a clickbait headline. The authors propose a system that classifies clickbait spoilers into three types (phrase, passage, multi) and trains specialized models for each, finding that this approach significantly outperforms a single, mixed model. Their work highlights the difficulty of the "passage" spoiler type and offers insights into improving clickbait spoiling generation.

What Problem Does This Paper Address?

Clickbait headlines are designed to arouse curiosity without revealing the core information, enticing users to click on a link. This often leads to wasted time for users who discover the content isn't what they expected or that the information could have been conveyed more directly. The task of "clickbait spoiling" aims to automatically generate a concise text that satisfies this induced curiosity, effectively acting as a summary that saves the user from needing to click through. Previous work suggested that classifying spoiler types is crucial for effective generation. This paper investigates this further by focusing on three specific spoiler types: "phrase" (short entities), "passage" (longer text segments), and "multi" (multiple disjointed spans), and explores how to best generate spoilers for each.

Methods

The core of the proposed approach involves a two-stage process: first, classifying the clickbait into one of the three spoiler types ("phrase," "passage," or "multi"), and second, using a type-specific model to generate the spoiler.

  • Spoiler Type Classification: A DeBERTa model, pre-trained on SQuAD, was fine-tuned for this task. They incorporated absolute position embeddings after the transformer layers to improve accuracy, achieving a classifier accuracy of around 0.76.
  • Type-Specific Spoiler Extraction:
    • Phrase Extractor: This model required minimal adjustments as it closely resembled the standard SQuAD task. It uses cross-entropy loss and a simple post-processing step to select the top non-empty span.
  • Passage Extractor: This model deals with longer and more variable answer spans. It uses a combination of cross-entropy loss and a contrastive loss to suppress negative logits. The post-processing involves a more complex reranking mechanism to select the best passage span.
  • Multi Extractor: This model handles spoilers composed of multiple disjointed text spans. It incorporates paragraph ID information and uses a "multi-softmax" to avoid suppressing multiple correct positive logits. The post-processing involves a greedy clustering algorithm to group similar answer spans and select the best ones.
  • Mixed Model: For comparison, a single "mixed" model was also trained to handle all spoiler types without prior classification.

The authors also detail a "Dataset Repair" process to correct inconsistencies in the provided dataset's span offsets and ground truth text.

Datasets

The primary dataset used is the Webis Clickbait Spoiling Corpus 2022. The paper mentions that the SQuAD dataset was used for pre-training the baseline DeBERTa QA model.

Key Insights and Ideas

  • Type-Specific Models are Superior: The most significant finding is that training separate models for each spoiler type ("phrase," "passage," "multi") yields better results than a single mixed model. This suggests that the distinct characteristics of each spoiler type require specialized handling.
  • "Passage" is the Toughest Nut to Crack: The "passage" spoiler type proved to be the most challenging. Its data is described as "noisiest," with less clear definitions and a tendency to be misclassified by the models. This "passage" type seems to act as a "mixture" of other types, making it difficult to generalize.
  • Classification is Key (with caveats): While a classifier with 0.76 accuracy was developed, the paper emphasizes that the benefit of type-specific models is contingent on having a good classifier. If the classifier is poor, a mixed model might be a better option.
  • Dataset Shift Matters: A noticeable performance gap between validation and test datasets was observed, attributed to a distribution shift, particularly a higher proportion of "passage" examples in the validation set.

Key Results

  • Improved Performance with Type-Specific Models: Separating models for each spoiler type led to improvements in BLEU-n and meteor scores across all types compared to the mixed model. The "multi" spoiler extractor, despite having the fewest examples and being the most dissimilar to standard QA tasks, achieved the largest improvement.
  • Classifier Accuracy: The spoiler type classifier achieved an accuracy of 0.76.
  • "Passage" Model Performance: The "passage" model showed the closest performance to the mixed model, reinforcing its difficulty.
  • Thresholding for Two-Stage Prediction: A two-stage prediction system using confidence thresholds for each spoiler type (phrase=0.75, passage=0.45, multi=0.7) was evaluated, showing a balanced performance.

Limitations and Future Work

  • Dataset Size and Splitting: The small dataset size and random splitting contributed to a distribution shift between validation and test sets, impacting performance.
  • Multi-Task Learning: Due to GPU memory limitations, multi-task learning (where models share encoder layers) was not experimented with, but it's hypothesized to be beneficial.
  • Metric Limitations: The paper discusses limitations of BLEU-n and meteor metrics, particularly their sensitivity to substrings and their differing ways of penalizing predictions. They suggest further study into designing more suitable metrics.
  • "Passage" Model Generalization: The "passage" model's specialization for its own type negatively impacted its performance on a general task. Designing a model that generalizes well to the "passage" type, despite its inherent noise, remains a challenge.
  • Further Exploration of Metrics: The authors suggest exploring metrics like BERTScore to reduce "pseudo mistakes" caused by metrics that might not fully capture semantic similarity.

Summary updated

BibTeX
@inproceedings{tang2023brooke,
  author = {Shirui Tang},
  title = {{Brooke-English at SemEval-2023 Task 5: Clickbait Spoiling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure