Jack-flood at SemEval-2023 Task 5:Hierarchical Encoding and Reciprocal Rank Fusion-Based System for Spoiler Classification and Generation

Sujit Kumar, Aditya Sinha, Soumyadeep Jana, Rahul Mishra, Sanasam Ranbir SinghACL 2023

TL;DR

This paper introduces a novel hierarchical encoding and reciprocal rank fusion approach for classifying and generating spoilers for clickbait content. Their system effectively identifies spoiler types and retrieves semantically relevant spoilers, addressing the challenge of clickbait's deceptive nature.

What Problem Does This Paper Address?

The proliferation of social media has led to an explosion of clickbait content, designed to grab user attention through sensational headlines. While clickbait is effective at driving engagement, it often misleads users about the actual content. This paper tackles the problem of "clickbait spoiling," which aims to generate or retrieve content that satisfies user curiosity without requiring them to click on the misleading post. Specifically, it addresses two subtasks: classifying the type of spoiler needed (phrase, passage, or multipart) and generating or retrieving that spoiler. This is crucial for mitigating the spread of misinformation and reducing user over-reliance on sensationalized content.

Methods

The authors propose a two-pronged approach:

1. Spoiler Type Classification: * Hierarchical Encoding (RPE Model): They leverage a "Recurrence over Pretrained Encoding" (RPE) model. This model first encodes individual sentences within a target paragraph using pretrained language models (like BERT, RoBERTa, DeBERTa). Then, a BiLSTM layer captures sequential dependencies between these sentence encodings. * Feature Engineering: Beyond the hierarchical encoding, they incorporate document length and count overlap features. These features are important because spoiler types often correlate with length (e.g., phrase spoilers are short, passage spoilers are longer). * Classification Approaches: Two methods are used: document classification (concatenating features) and a similarity-based approach (estimating angle and difference between encoded representations). The similarity-based approach, combined with features, proved superior.

2. Spoiler Generation: * Passage Spoiler Retrieval: They combine multiple information retrieval techniques. Initially, BM25 is used for keyword-based retrieval. This is augmented by semantic similarity scores derived from Sentence-BERT and DeBERTa. Finally, Reciprocal Rank Fusion (RRF) is employed to combine the rankings from BM25, S-BERT, DeBERTa, and a MonoT5 model, selecting the top-ranked sentence as the passage spoiler. * Phrase Spoiler Retrieval: This is framed as a question-answering task. They use pretrained RoBERTa and DeBERTa models trained on QA tasks, with RoBERTa alone showing the best performance. * Multipart Spoiler Retrieval: This approach combines the passage and phrase retrieval methods. They first retrieve top-k passage spoilers and then use RoBERTa's QA capabilities on these passages to generate multipart spoilers.

Datasets

The study utilizes a dataset curated by Hagen et al. (2022) for the SemEval-2023 Task 5: Clickbait Spoiling. The paper notes that the dataset exhibits an imbalance, with the "multipart" class being over-represented compared to "phrase" and "passage" classes. No additional data was used for training.

Key Insights and Ideas

  • Hierarchical Structure Matters: The success of the RPE model highlights the importance of considering the hierarchical structure of target paragraphs (sentences within paragraphs) for spoiler classification.
  • Semantic Similarity is Key: The similarity-based approach for classification outperformed document classification, suggesting that the semantic relationship between the clickbait post and the spoiler is more critical than just feature concatenation.
  • RRF for Robust Retrieval: Combining multiple retrieval methods (BM25, S-BERT, DeBERTa, MonoT5) via Reciprocal Rank Fusion significantly improves passage spoiler retrieval, demonstrating the power of ensemble methods in IR.
  • BERTScore as a Better Metric: The authors argue that BERTScore is a more suitable evaluation metric for spoiler generation than BLEU-4 or METEOR, as it better captures semantic similarity and meaning preservation, even when lexical overlap is low.

Key Results

  • Spoiler Classification: The RPE model with a similarity-based approach and the inclusion of count overlap and document length features (RPE(RoBERTa, S, F)) achieved the best performance for spoiler type classification.
  • Spoiler Generation: The system demonstrated effectiveness in retrieving spoilers that are semantically very close to the ground truth spoilers, as indicated by BERTScore. While BL4 and MET scores were average, BERTScore showed strong performance, suggesting the generated spoilers are semantically meaningful. For instance, in Table 2, the BERTScore for "All" categories is consistently high (around 0.85-0.89).

Limitations and Future Work

  • Dataset Imbalance: The dataset's imbalance in spoiler types (multipart over-represented) could potentially bias model performance.
  • Average BL4/MET Scores: While BERTScore is strong, the average performance on BL4 and METEOR metrics for spoiler generation suggests room for improvement in token-level accuracy.
  • Future Work: The authors plan to explore extractive summarization methods for future spoiler generation tasks.

Summary updated

BibTeX
@inproceedings{kumar2023jack,
  author = {Sujit Kumar and Aditya Sinha and Soumyadeep Jana and Rahul Mishra and Sanasam Ranbir Singh},
  title = {{Jack-flood at SemEval-2023 Task 5:Hierarchical Encoding and Reciprocal Rank Fusion-Based System for Spoiler Classification and Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure