Abstract
We introduce a model of coherence which captures the intentional discourse structure in text. Our work is based on the hypothesis that syntax provides a proxy for the communicative goal of a sentence and therefore the sequence of sentences in a coherent discourse should exhibit detectable structural patterns. Results show that our method has high discriminating power for separating out coherent and incoherent news articles reaching accuracies of up to 90%. We also show that our syntactic patterns are correlated with manual annotations of intentional structure for academic conference articles and can successfully predict the coherence of abstract, introduction and related work sections of these articles.
Introduction
In Natural Language Processing (NLP), "coherence" is the property that makes a text a unified whole rather than a random collection of sentences. Traditionally, researchers have modeled this through entity-based methods (tracking mentions of specific objects) or content-based methods (tracking topic shifts). However, these approaches often overlook the intentional structure—the underlying purpose of each sentence, such as providing motivation, describing a method, or presenting results.
Manually labeling these intentions is labor-intensive and expensive. This paper proposes an unsupervised model that captures intentional structure by using syntax as a "free" signal, based on the premise that the "flow" of a document can be modeled through detectable syntactic patterns.
Methodology
The core hypothesis is that specific communicative goals have distinct "syntactic fingerprints." For instance, a definition might consistently follow a specific noun-phrase and relative-clause structure. The authors propose two syntactic representations:
- Productions: Utilizing formal context-free grammar (CFG) rules from a sentence's parse tree.
- d-sequences: A novel approach that truncates a parse tree at a specific depth (d). This captures the high-level "skeleton" of a sentence while abstracting away from specific vocabulary.
These representations are used in two model types: a Local Co-occurrence Model, which calculates the probability of adjacent syntactic structures, and a Syntax-HMM (Hidden Markov Model). The HMM automatically clusters sentences with similar syntax into "hidden states" (representing communicative goals) and learns the typical transitions between these states to model document flow.
Datasets and Evaluation
The models were evaluated across several diverse domains:
- News Corpora: Reports on airplane accidents (NTSB) and earthquakes (Associated Press), standard benchmarks for coherence tasks.
- ART Corpus: Chemistry journal articles manually annotated with "argumentative zones" (e.g., Hypothesis, Goal, Result).
- ACL Anthology Network (AAN): Computational linguistics papers used to test if the model could distinguish between high-level conference papers and more preliminary workshop papers.
Key Results and Insights
The Syntax-HMM demonstrated significant discriminating power, reaching accuracies between 82% and 95% on sentence ordering tasks in news corpora, far exceeding the random baseline.
Key insights from the evaluation include:
- Syntax as a Proxy for Intent: The structural arrangement of parts of speech is often sufficient to identify a sentence's role (e.g., background statement vs. conclusion) without semantic knowledge.
- Complementary Signals: Combining syntactic models with traditional entity-grid models improved performance, indicating that syntax captures a unique dimension of writing quality that lexical models miss.
- Validation via Human Labels: On the ART corpus, unsupervised HMM clusters correlated strongly with human labels. For example, one cluster identified "Background" sentences with 64.9% accuracy.
Conclusion
This work demonstrates that syntactic patterns are a powerful, unsupervised signal for modeling discourse coherence. While the model relies on high-quality parsing and is somewhat genre-dependent, it successfully distinguishes well-organized text from scrambled versions and identifies the stable intentional structures characteristic of high-quality academic publications. Future work will explore integrating entity, topic, and syntactic dimensions into a unified model of text quality.
Summary updated