On the Role of Context for Discourse Relation Classification in Scientific Writing

Stephen Wan, Wei Liu, Michael Strube

Abstract

With the increasing use of generative Artificial Intelligence (AI) methods to support science workflows, we are interested in the use of discourse-level information to find supporting evidence for AI-generated scientific claims. A first step towards this objective is to examine the task of inferring discourse structure in scientific writing. In this work, we present a preliminary investigation of pretrained language model (PLM) and Large Language Model (LLM) approaches for Discourse Relation Classification (DRC), focusing on scientific publications, an under-studied genre for this task. We examine how context can help with the DRC task, with our experiments showing that context, as defined by discourse structure, is generally helpful. We also present an analysis of which scientific discourse relation types might benefit most from context.

Introduction

The rise of "AI for Science" has introduced powerful tools for research planning and question answering, such as Google’s Co-scientist and OpenAI’s Deep Research. However, a significant hurdle remains: the problem of hallucinations in generative models. To trust an AI-generated scientific claim, researchers require verifiable supporting evidence from source literature.

A key component of finding this evidence is Discourse Relation Classification (DRC). DRC involves identifying how fragments of text—Elementary Discourse Units (EDUs)—relate to one another logically (e.g., is one a result of another, or a condition?). While well-studied in news text, scientific writing is denser and more complex. Most current models analyze EDU pairs in isolation, ignoring the broader context. This paper investigates whether providing context informed by a paper's discourse structure improves a model's ability to classify these relations.

Methodology

The authors compared two main modeling approaches:

  1. PLM-Finetuning (RoBERTa): A version of RoBERTa trained specifically to predict discourse relations.
  2. LLM-based Inference (GPT-4 and LLaMA 3.1 70B): Using In-Context Learning (prompting) to classify relations without specific fine-tuning.

Three Context Selection Schemes were tested:

  • Default: No context; only the target EDU pair.
  • Add-n (ADn): A sliding window providing the $n$ preceding sentences.
  • Oracle-n (ORn): Using the ground-truth discourse tree to find the logical "parent" unit of the fragment. This "structured context" provides the text the fragment is logically attached to, regardless of physical adjacency.
Figure 1: Example of a discourse dependency tree for a scientific abstract, illustrating logical relations like Background, Comparison, and Findings.

Datasets

The study utilized two major discourse datasets for scientific text:

  • SciDTB: Focused on the Natural Language Processing (NLP) domain.
  • CovDTB: Focused on the Biomedical domain (COVID-19 articles).

Both datasets represent discourse as dependency trees, where nodes are text units and edges are labeled relations.

Key Results and Discussion

The Power of Structured Context

The experiments revealed that physical proximity does not equate to logical relevance. Simply adding the preceding sentence (AD1) often provided negligible gains or even decreased performance. However, when models were given the "Oracle" context (OR1)—the actual logical parent in the discourse tree—performance improved significantly.

Figure 4: Macro-F1 results for the fine-tuned RoBERTa model across different context schemes.

LLM vs. PLM Performance

The impact of structured context was most dramatic for Large Language Models. For GPT-4 on the SciDTB dataset, Macro-F1 scores jumped from 22.06 (Default) to 52.61 (OR1). Despite these gains, the smaller, fine-tuned RoBERTa model often maintained a performance edge in specific categories, reaching a Macro-F1 of 58.33 on SciDTB with structured context.

Domain and Relation Analysis

Models generally performed better on the Biomedical dataset (CovDTB) than the NLP dataset (SciDTB), likely due to more homogenous writing conventions in biomedicine. Context was particularly helpful for relations that are "explicitly marked" by connectives (e.g., however, because), as it helps the model identify the connective's role within the argument flow.

Conclusion

This work demonstrates that structured context, derived from discourse trees, is a vital component for accurate Discourse Relation Classification in scientific writing. While the current study relies on ground-truth "Oracle" trees, future work will explore automatically inferred graph representations. These insights are a crucial step toward building AI tools that can accurately cite source material and provide faithful, evidence-based justifications for scientific claims.

Summary updated

Abstract

With the increasing use of generative Artificial Intelligence (AI) methods to support science workflows, we are interested in the use of discourse-level information to find supporting evidence for AI generated scientific claims.A first step towards this objective is to examine the task of inferring discourse structure in scientific writing.In this work, we present a preliminary investigation of pretrained language model (PLM) and Large Language Model (LLM) approaches for Discourse Relation Classification (DRC), focusing on scientific publications, an under-studied genre for this task.We examine how context can help with the DRC task, with our experiments showing that context, as defined by discourse structure, is generally helpful.We also present an analysis of which scientific discourse relation types might benefit most from context.Recent Artificial Intelligence (AI) advances coupled with the agentic AI approach have seen a burst of activity in the area of "AI for Science", the application of AI techniques to help accelerate scientific discovery.Examples include usage of Google's Co-scientist (Penads et al., 2025), OpenAI's Deep Research 1 , NVidia's foundation models for life sciences 2 and the agentic AI platform Future House 3 .Many of these tools offer an AI research assistant that helps complex research information needs, such as question answering and research planning.Within these AI for Science applications, generative AI approaches based on Large Language Models (e.g., Brown et al., 2020) are used to generate answers (novel text) to complex questions, introducing the problem of addressing hallucina-
BibTeX
@misc{wan2025role,
  author = {Stephen Wan and Wei Liu and Michael Strube},
  title = {{On the Role of Context for Discourse Relation Classification in Scientific Writing}},
  year = {2025},
  eprint = {2510.26354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.26354},
}
Figure