A Joint Model for Structure-based News Genre Classification with Application to Text Summarization

Zeyu Dai, Ruihong HuangACL 2021 · OpenReview

TL;DR

This paper introduces a novel joint model that simultaneously classifies news article structures and identifies specific news elements within them. This integrated approach significantly improves both tasks by leveraging their inherent dependencies, leading to better text summarization performance, especially for challenging news genres like "Kabob."

What Problem Does This Paper Address?

Journalists often structure news articles in predictable ways to convey information effectively. For instance, the "Inverted Pyramid" structure prioritizes the most crucial information first. However, other structures exist, like "Kabob," which uses a narrative hook before the main story, posing challenges for automated analysis. Recognizing these structures and the specific "news elements" (like ledes, synopses, or body sections) that compose them is crucial for downstream NLP tasks such as text summarization, information extraction, and text segmentation. Previous work often treated these tasks independently or used pipeline approaches, which can propagate errors. This paper aims to bridge this gap by developing a unified model that jointly learns both news structure types and news element sequences.

Methods

The core of this work is a joint neural network model that processes an entire news article to predict two things simultaneously:

  1. Document-level News Structure Type: Classifying the article into one of four predefined structures (Inverted Pyramid, Martini Glass, Kabob, Narrative).
  2. Paragraph-level News Element Tags: Assigning a tag to each paragraph indicating which news element it belongs to (e.g., Standard Lede, Image Lede, Synopsis, Narration, Body Section).

The model employs a hierarchical BiLSTM architecture. It first uses a word-level BiLSTM to refine word representations, incorporating features like character embeddings, GloVe, ELMo, or BERT embeddings. This is followed by a paragraph-level BiLSTM that builds paragraph representations. For document-level structure prediction, a max-pooling operation over paragraph representations creates a document representation, which is then fed into a softmax classifier. For paragraph-level element tagging, a linear-chain Conditional Random Field (CRF) layer is used to model label dependencies and ensure coherent sequences. The model is trained by minimizing a combined loss function that includes both document-level and paragraph-level losses.

Datasets

The primary dataset used is the News Genre dataset, created by the authors in previous work (Dai et al., 2018). This dataset contains 853 English news articles annotated with both document-level news structure labels and paragraph-level news element tags. For extrinsic evaluation, the authors used the CNN/DailyMail dataset, a widely used benchmark for text summarization.

Key Insights and Ideas

  • Interdependence is Key: The paper strongly supports the idea that news structure and news element identification are not independent tasks. Jointly modeling them preserves crucial two-way dependencies, leading to better predictions for both.
  • Kabob is Tricky: The "Kabob" news genre, which delays the main story, presents a significant challenge for text summarization. Identifying its specific structure and elements is shown to be particularly beneficial for improving summarization performance on this genre.
  • Beyond Structure Type: While predicting the overall news structure is valuable, identifying the specific sequence of news elements within that structure provides finer-grained information that can further enhance NLP tasks.

Key Results

  • Intrinsic Evaluation: The joint model consistently outperforms its single-task variants and pipeline approaches across various word embedding strategies (GloVe, ELMo, BERT). For example, the joint model with ELMo embeddings achieved 80.0% accuracy and 56.0% macro F1-score for document-level news structure prediction, and 78.3% accuracy and 53.2% macro F1-score for paragraph-level news element tagging.
  • Extrinsic Evaluation (Text Summarization): Incorporating the predicted news element tags into a BERT-based summarization system improved ROUGE scores by 0.17 (R-1), 0.04 (R-2), and 0.11 (R-L). Adding news structure types further boosted performance.
  • Genre-Specific Improvements: For the challenging "Kabob" genre, the proposed tags led to the largest improvements in ROUGE scores (0.37 on R-1, 0.14 on R-2, and 0.34 on R-L), highlighting the model's effectiveness in handling difficult cases.

Limitations and Future Work

  • Minority Class Performance: The authors acknowledge that performance on minority classes of news structures and elements (e.g., "Narration") could be improved.
  • Distinguishing Ledes: The model sometimes struggles to differentiate between "Standard Lede" and "Image Lede," suggesting that incorporating information about key events could be beneficial.
  • Narrative Style Identification: Identifying "Narration" paragraphs, which are inherently narrative, is noted as a challenging task.
  • Future Directions: The authors plan to explore semi-supervised learning to improve performance on minority classes and investigate the application of their news genre models to other NLP tasks like text quality assessment and information extraction.

Summary updated

BibTeX
@inproceedings{dai2021joint,
  author = {Zeyu Dai and Ruihong Huang},
  title = {{A Joint Model for Structure-based News Genre Classification with Application to Text Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=pmyNAIgscZc},
}
Figure