Joint Modeling of Entities and Discourse Relations for Coherence Assessment

Wei Liu, Michael StrubeEMNLP 2025

TL;DR

This paper introduces novel methods for jointly modeling entities and discourse relations to improve text coherence assessment. By integrating these two crucial linguistic aspects, the authors demonstrate significant performance gains over models that consider either feature in isolation, leading to more robust and accurate coherence evaluation.

What Problem Does This Paper Address?

Text coherence, the property that makes a text flow logically and be easy to understand, is crucial for many Natural Language Processing (NLP) tasks. Traditionally, coherence has been approached from two main perspectives: entity-based coherence (tracking how entities are introduced and maintained) and discourse relation-based coherence (understanding the logical or rhetorical links between sentences). While both are important, most existing work focuses on one or the other, neglecting their interplay. This paper addresses the gap by proposing methods that jointly model these two aspects, arguing that their combined effect is essential for a comprehensive understanding of text coherence.

Methods

The authors propose two main approaches:

  1. Fusion Transformer: This method first identifies entities and discourse relations within a document. These elements, along with the sentences themselves, are then organized into a flat structure. This structure is fed into a modified Transformer model that incorporates a novel "position-aware attention" mechanism and a "visible matrix" to guide attention between related elements. This allows the model to learn from the combined signals of entities, relations, and sentence order.
  2. Prompt-based LLM: This approach leverages the generative capabilities of Large Language Models (LLMs). Instead of building a dedicated fusion module, it describes the connections between sentences, entities, and discourse relations in natural language. This information is then included in a prompt given to an LLM, instructing it to assess coherence based on this rich contextual input. This method aims to utilize LLMs more effectively by providing them with explicit linguistic cues.

Datasets

The authors evaluate their methods on three widely used corpora for coherence modeling:

  • GCDC: A corpus of real-world texts from diverse domains (forums, emails, reviews) annotated with coherence scores from 1 to 3 (low, medium, high).
  • CoheSentia: A dataset of stories generated by GPT-3, annotated by humans with coherence scores from 1 to 5. The authors group scores to address data imbalance.
  • TOEFL: A dataset originally for automated essay scoring, but commonly used for coherence evaluation, with essays rated as low, medium, or high coherence.

Key Insights and Ideas

  • Synergy of Entities and Relations: The most significant insight is that jointly modeling entities and discourse relations leads to substantially better coherence assessment than using either feature alone. This highlights that coherence is not just about tracking who or what is being discussed, nor solely about the logical flow, but a complex interplay of both.
  • Robustness to Imbalanced Data: The proposed methods, especially when jointly modeling features, show improved robustness to imbalanced datasets. They help mitigate the bias towards majority classes (e.g., "high" coherence) and improve performance on underrepresented classes (e.g., "low" coherence).
  • Domain-Agnostic Features: Entity and discourse relation patterns act as effective domain-agnostic features, enabling better transferability of models across different domains.
  • LLMs as Coherence Assessors: The prompt-based approach demonstrates that LLMs can effectively leverage explicit linguistic information (entities and relations) for coherence assessment, even in zero-shot settings, suggesting their potential for sophisticated linguistic understanding.

Key Results

  • Fusion Transformer: The proposed "Our Method I" (Fusion Transformer) consistently outperformed strong baselines (TextOnly, TextEnty, TextRel) across all three datasets. For instance, on GCDC, it achieved a 1-2% improvement over baselines, and approximately a 3% gain on CoheSentia.
  • Prompt-based LLM: In zero-shot settings, incorporating entity and discourse relation information significantly boosted LLM performance. On GCDC, gains of over 6 points were observed compared to the TextOnly baseline.
  • TOEFL Performance: On the TOEFL dataset, the Fusion Transformer achieved an average accuracy of 78.66%, outperforming previous state-of-the-art models. The prompt-based method also showed significant improvements.
  • Case Studies: Qualitative analysis showed that models incorporating both entity and discourse relations were better at correctly identifying low-coherence texts, especially when entity cues were missing or ambiguous.

Limitations and Future Work

The authors acknowledge several limitations:

  • PDTB Parser Imperfection: The performance of the discourse parser used is not perfect, and future work could explore more powerful parsers, potentially LLM-based ones, for better discourse relation analysis.
  • Limited Discourse Relation Framework: The study is limited to PDTB-style discourse relations. Extending the analysis to other frameworks like RST (Rhetorical Structure Theory) could offer broader insights.
  • LLM Scale: Experiments were primarily conducted with Llama-8B and Llama-3.3-70B. Evaluating with even larger or different LLMs could reveal further capabilities.

Future work could focus on developing more robust discourse parsers, exploring alternative discourse frameworks, and evaluating the proposed methods with a wider range of LLMs.

Summary updated

BibTeX
@inproceedings{liu2025joint,
  author = {Wei Liu and Michael Strube},
  title = {{Joint Modeling of Entities and Discourse Relations for Coherence Assessment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure