TL;DR
This paper introduces a novel metric for automatically evaluating discourse coherence by leveraging the outputs of coreference resolution models. Unlike previous methods that focus on local sentence transitions, this metric captures longer-range relationships between discourse entities, leading to improved performance in distinguishing coherent from incoherent texts.
What Problem Does This Paper Address?
Evaluating discourse coherence is crucial for various Natural Language Processing (NLP) tasks like text generation, summarization, and automated essay scoring. Traditional approaches often rely on linguistic theories like Centering Theory to model how discourse entities (like nouns and pronouns) transition between sentences. However, these methods can struggle with languages like Japanese, where frequent ellipses make it difficult to explicitly track entity transitions. Furthermore, existing "entity grid" models primarily focus on local coherence, analyzing transitions between adjacent sentences, potentially missing coherence cues that span longer distances. This paper aims to address this by proposing a metric that can capture these longer-range coherence relationships.
Methods
The core of this paper is a new metric for discourse coherence that utilizes the outputs of a coreference resolution model. The underlying assumption is that writers tend to use coreference relations more appropriately in coherent texts. The metric is calculated in two steps:
- Coreference Resolution: A trained coreference resolution model is applied to the target text.
-
Coherence Score Calculation: The coherence score is computed as the average of the reliability scores of identified coreference relations. Specifically, for each anaphor (a word referring to something previously mentioned), the metric considers the most likely antecedent (the entity it refers to) and uses the confidence score (e.g., predicted probability) from the coreference model for that pair. The formula is:
$ \text{coherence}(T) = \frac{1}{N} \sum_{i,j} \text{score}_{\text{ana}}(i, j) $
where $N$ is the number of candidate anaphors, $j$ is an anaphor, $i$ is its most likely antecedent, and $ \text{score}_{\text{ana}}(i, j) $ is the reliability score of the coreference relation.
For their coreference resolution model, the authors employ an ILP-based approach that optimizes for Japanese anaphora resolution, which they adapt to provide the necessary reliability scores.
Datasets
The primary dataset used for evaluation is the NAIST text corpus. This corpus consists of Japanese newspaper articles with manually annotated NP coreference relations. The authors split the corpus into training and testing sets, following previous work. For the information ordering task, they created "incoherent texts" by randomly scrambling the sentences of original "coherent texts" to create pairs for evaluation.
Key Insights and Ideas
- Coreference as a Coherence Signal: The paper strongly suggests that the appropriateness and reliability of coreference resolution are direct indicators of discourse coherence. This moves beyond simply tracking entity presence to evaluating the quality of their connections.
- Long-Range Coherence: The proposed metric's ability to capture relationships between entities that are not necessarily adjacent in text is a significant advantage over local coherence models like the entity grid. This is crucial for understanding how different parts of a text connect, even across several sentences.
- Cross-Lingual Applicability (Implicit): While the focus is on Japanese, the core idea of using coreference resolution outputs for coherence evaluation is potentially applicable to other languages, especially those with rich coreference phenomena.
- Coreference Model Performance Correlates with Coherence: Experiment 1 shows that coreference resolution models perform better on coherent texts than on incoherent ones, reinforcing the central hypothesis.
Key Results
- Improved Performance on Information Ordering: The proposed metric achieved better accuracy than the standard entity grid model on the pairwise information ordering task (76.1% vs. 70.7% accuracy).
- Synergistic Effect: When the proposed metric was used as a feature in addition to the entity grid model, the accuracy further improved to 78.2%, demonstrating the complementary nature of the approaches.
- Coreference Resolution Performance: The coreference resolution model itself showed better performance (higher F-score) on coherent texts (0.560) compared to incoherent texts (0.516 ± 0.004), supporting the premise that coherence influences coreference.
Limitations and Future Work
- Sparsity of Coreference Relations: The authors acknowledge that coreference relations can sometimes be sparse in a text, which might lead to an inappropriate score.
- Future Directions: To address sparsity and enhance coherence modeling, they suggest incorporating other reference behaviors like zero anaphora and bridging anaphora. They also propose exploring latent topic transitions as another key factor for capturing text coherence.
Summary updated