TL;DR
This paper presents a novel approach to automatically identifying the discourse function of paragraphs in news articles, significantly improving upon previous methods. By leveraging a linear chain Conditional Random Field (CRF) model and a rich set of linguistic features, the authors achieve state-of-the-art performance, offering a more nuanced understanding of news article structure.
What Problem Does This Paper Address?
Understanding the structure of written text is crucial for effective information retrieval and analysis. For news articles, in particular, there's a well-established hierarchical theory of discourse proposed by van Dijk (1988). This theory posits that paragraphs within a news article serve specific functions, contributing to the overall narrative and commentary. These functions can be broadly categorized into those relating to the main storyline (like "MAIN EVENTS" or "CONSEQUENCES") and those providing commentary (like "VERBAL REACTIONS" or "EVALUATIONS").
The challenge lies in automatically identifying these discourse functions for each paragraph. Previous work, notably by Yarlott et al. (2018), demonstrated the feasibility of this task using a Support Vector Machine (SVM) model and a limited set of linguistic features, achieving an F₁ score of 0.54. However, this approach did not fully capture the interdependencies between paragraphs and their discourse functions, which are crucial for a comprehensive understanding of news structure. This paper aims to bridge this gap by developing a more sophisticated model that can better model these sequential dependencies.
Methods
The core of this paper's contribution lies in its improved methodology for discourse function identification. The authors employ a linear chain Conditional Random Field (CRF) model. Unlike previous generative models like Hidden Markov Models (HMMs) that assume independence between observations, CRFs are discriminative models that can capture complex dependencies between sequential data. This is particularly beneficial here, as the discourse function of a paragraph is likely influenced not just by the preceding paragraph but by a longer sequence of previous paragraphs.
The CRF model is enhanced with a comprehensive set of feature functions. These functions are divided into two main categories:
- Discourse Labels' Order (Hj): This component models the sequential dependencies between discourse labels. It uses a binary matrix derived from the van Dijk theory to represent which labels are likely to follow others.
- Discourse Labels' Content (SFj): This component captures the linguistic characteristics of paragraphs that are indicative of their discourse function. The authors build upon features used in prior work (bag-of-words, tf-idf, paragraph vectors, previous paragraph label) and introduce several new ones:
- Lexical: Bigrams to capture language patterns.
- Positional: Paragraph size (number of sentences) and position within the document.
- Syntactic: Part-of-Speech (POS) tag counts.
- Semantic: Features related to reported speech, majority event tense, subevent relations, and Named Entity Recognition (NER) vectors.
The model is trained using the L-BFGS method to estimate the weights of these feature functions.
Datasets
The authors utilize a gold-standard corpus previously developed by Yarlott et al. (2018). This dataset is a subset of the Automated Content Extraction (ACE) Phase 2 corpus, a widely-used benchmark for news articles. The corpus consists of 50 news documents, totaling 28,236 words distributed across 644 paragraphs. Each paragraph is assigned a single discourse label according to van Dijk's theory. The dataset was doubly annotated to ensure reliability.
Key Insights and Ideas
- Interdependencies Matter: The most significant insight is the importance of modeling interdependencies between paragraphs and their discourse functions. A linear chain CRF, by its nature, excels at this, leading to substantial performance gains over models that treat paragraphs more independently.
- Rich Features Drive Performance: The introduction of a diverse set of features, particularly the semantic ones like reported speech and subevent relations, significantly boosts performance, especially for specific discourse labels.
- Reported Speech is Key for Verbal Reactions: The reported speech feature proved particularly impactful, leading to a high F₁ score for the "VERBAL REACTIONS" class, which is prevalent in the dataset.
- Syntactic and Positional Features Aid Specific Labels: For labels like "CONSEQUENCES," "HISTORY," "EXPECTATIONS," and "EVALUATIONS," syntactic and positional features were found to be most helpful.
- Scarcity vs. Prevalence: Performance on less frequent labels (e.g., "HISTORY," "EXPECTATIONS") was lower, but the model still showed improvement compared to prior approaches, indicating that the new features help capture their distinct language.
Key Results
The proposed CRF model achieved a micro-averaged F₁ score of 0.71, representing a significant 31.5% improvement over the previously best-performing SVM model (0.54 F₁). This performance gain is substantial and demonstrates the effectiveness of the chosen methodology and features.
The paper also presents per-label F₁ scores, highlighting strong performance on "LEAD" (0.95), "MAIN EVENTS" (0.69), "CIRCUMSTANCES" (0.72), and "VERBAL REACTIONS" (0.91). While performance on less frequent labels like "HISTORY" (0.24) and "EXPECTATIONS" (0.26) was lower, the model still showed improvements over baselines.
Limitations and Future Work
The authors acknowledge several limitations and suggest avenues for future research:
- Error Propagation: The reliance on automated systems for extracting features like events and subevent relations means that errors from these systems can propagate and affect the overall performance.
- Dataset Size: While the ACE Phase 2 corpus is a standard benchmark, a larger and more diverse corpus could further improve the model's generalization capabilities.
- Refined Event Detection: More sophisticated event detection solutions could lead to even better performance.
- Hierarchical Labeling: The paper notes that some previous models (like decision trees and random forests) did not leverage the full hierarchical nature of van Dijk's theory, only considering leaf labels. Future work could explore models that explicitly incorporate this hierarchy.
- Modeling Inter-Dependencies: The authors emphasize the importance of modeling paragraph and discourse label type inter-dependencies, suggesting that further exploration in this area could yield even greater improvements.
Summary updated