Abstract
Discourse structure is a key aspect of all forms of text, providing valuable information both to humans and machines. We applied the hierarchical theory of news discourse developed by van Dijk to examine how paragraphs operate as units of discourse structure within news articles—what we refer to here as document-level discourse. This document-level discourse provides a characterization of the content of each paragraph that describes its relation to the events presented in the article (such as main events, backgrounds, and consequences) as well as to other components of the story (such as commentary and evaluation). The purpose of a news discourse section is of great utility to story understanding as it affects both the importance and temporal order of items introduced in the text—therefore, if we know the news discourse purpose for different sections, we should be able to better rank events for their importance and better construct timelines. We test two hypotheses: first, that people can reliably annotate news articles with van Dijk’s theory; second, that we can reliably predict these labels using machine learning. We show that people have a high degree of agreement with each other when annotating the theory (F1 > 0.8, Cohen’s kappa > 0.6), demonstrating that it can be both learned and reliably applied by human annotators. Additionally, we demonstrate first steps toward machine learning of the theory, achieving a performance of F1 = 0.54, which is 65% of human performance. Moreover, we have generated a gold-standard, adjudicated corpus of 50 documents for document-level discourse annotation based on the ACE Phase 2 corpus.
Introduction
While human readers intuitively navigate the "architecture" of news articles, machines struggle to grasp the underlying discourse structure. Unlike chronological narratives, news stories often follow an "inverted pyramid" style, prioritizing critical information and then layering context, expert commentary, and historical background. Identifying the discourse function of a paragraph—the specific "job" it performs within the narrative—is essential for advanced NLP tasks like timeline construction and event importance ranking. This paper investigates whether Teun van Dijk’s hierarchical theory of news discourse can be reliably applied by humans and learned by machines.
Theoretical Framework: van Dijk’s Hierarchy
The researchers adopted the hierarchical theory proposed by van Dijk (1988), which organizes news reports into a tree structure. This framework distinguishes between the "Summary" (Headline and Lead) and the "Story," which further branches into "Situation" (Episode and Background) and "Comments" (Verbal Reactions and Conclusions).
By categorizing paragraphs into these functional units, the model captures the relationship between the primary events and the surrounding context or evaluative commentary.
Corpus Development and Human Annotation
To create a benchmark for this task, the authors annotated a subset of the ACE Phase 2 corpus. The resulting gold-standard dataset consists of 50 documents, encompassing 28,236 words across 644 paragraphs. Three Ph.D. students performed double-blind annotations, with an adjudicator resolving conflicts to create an adjudicated corpus.
The annotation process revealed that humans can achieve high reliability (F1 > 0.75, κ > 0.60), validating van Dijk’s theory as a robust framework for modern news analysis. The dataset is notably rich in "Verbal Reactions" and "Circumstances," reflecting the quote-heavy nature of professional journalism.
Machine Learning Results
The study evaluated several classifiers, including Support Vector Machines (SVM), Decision Trees, and Random Forests. Features included Bag of Words, TF-IDF, Paragraph Vectors (Doc2Vec), and sequential features (the label of the preceding paragraph).
The SVM classifier performed best, achieving an overall F1 score of 0.54. While this represents approximately 65% of human performance, results varied significantly across categories:
- Lead (F1: 0.87): Highly predictable due to its consistent position at the start of articles.
- Verbal Reactions (F1: 0.76): Easily identified by lexical cues like quotation marks and attribution verbs (e.g., "said").
- Main Events (F1: 0.23): Surprisingly difficult to predict, likely due to high linguistic diversity.
- History (F1: 0.05): The most challenging category, suffering from data scarcity and a lack of distinct markers.
Conclusion and Future Directions
This work demonstrates that document-level discourse annotation is a viable task for both humans and machines. Future research will focus on exploiting the hierarchical nature of the labels rather than treating them as a flat list. Additionally, incorporating more sophisticated features like coreference chains and expanding the corpus to better represent rare categories could significantly bridge the gap between machine and human performance.
Summary updated