It Depends on the Task: Rethinking Rhetorical Segmentation of Dutch Judicial Decisions

Samyak Sheth, Gerasimos Spanakis, Gijs van DijckarXiv 2026 · arXiv

TL;DR

This paper introduces the first sentence-level annotated dataset for rhetorical role classification in Dutch judicial decisions. The authors demonstrate that there is no "one-size-fits-all" model; instead, the optimal classification strategy—whether a direct multi-class approach or a staged pipeline—depends heavily on whether a downstream task prioritizes precision (like outcome prediction) or recall (like extractive summarization).

What Problem Does This Paper Address?

In the legal domain, knowing the "rhetorical role" of a sentence—whether it describes a fact, a party's submission, the court's reasoning, or the final ruling—is essential for high-performance NLP. However, judicial text is messy. Unlike a scientific paper with clear "Methods" or "Results" sections, Dutch judicial decisions often interleave these roles within a single passage. A sentence's position or section header isn't always a reliable indicator of its function.

The authors tackle two main gaps:

  1. Data Scarcity: There was previously no publicly available, sentence-level annotated corpus for Dutch case law.
  2. Strategy Evaluation: Most existing work treats role classification as a standard multi-class problem. This paper investigates whether specialized pipelines or Bayesian updates (using section headers as priors) can better handle the nuances of legal text.

Methods

The researchers evaluated several model families, including a lexical baseline (Logistic Regression with TF-IDF), Dutch encoder-based models (RobBERT and BERTje), and several instruction-tuned LLMs (Gemma, Qwen) in zero/few-shot settings.

They compared four classification formulations:

  • Five-way Classification: A single model predicting all roles plus a "None" category.
  • Pipeline A: A binary "None Detector" followed by a four-way role classifier.
  • Pipeline B: A "None Detector" followed by independent One-vs-Rest (OvR) detectors for each role.
  • Bayesian Updating: A novel post-processing step that adjusts the model's text-based probability using a "header prior" (how likely a role is to appear under a specific section header).
Figure 2. Overview of classification formulations, end-to-end pipeline combinations, and Bayesian updating. Green boxes indicate classification components; blue boxes indicate final five-way predictions; grey boxes indicate inputs.

Datasets

The authors created a new experimental corpus of 100 Dutch judicial decisions (2015–2021) covering civil, criminal, and administrative law.

  • Size: 8,391 sentences.
  • Labels: materiele feiten (facts), proceshandelingen (submissions), beoordeling (reasoning), beslissing (decision), and None.
  • Quality: High inter-annotator agreement (mean Cohen’s $\kappa = 0.871$).
Table 1. Label distribution in the annotated dataset.

Key Insights and Ideas

  • "None" is the Hardest Part: Detecting whether a sentence has any legal role (None Detection) is significantly more difficult than distinguishing between the roles themselves. Neither Bayesian priors nor confidence thresholding significantly improved this stage.
  • The Precision-Recall Trade-off: Different downstream tasks have different needs. For example, extractive summarization needs high recall to ensure no facts are missed. Conversely, outcome prediction needs high precision to ensure the model isn't "hallucinating" a ruling based on noisy, non-decision sentences.
  • Headers are Signals, Not Rules: While section headers provide a strong hint, they are not definitive. The Bayesian update treats them as a probabilistic signal, which improved performance for role classification but proved less useful for filtering out "None" sentences.
  • Pipelines Expose Trade-offs: While a staged pipeline (Pipeline B) only modestly outperformed a direct five-way classifier in aggregate F1, its real value lies in modularity. It allows developers to tune specific components (e.g., a high-precision "Decision" detector) without retraining the entire system.

Key Results

  • Top Model: RobBERT consistently outperformed BERTje, lexical baselines, and prompting-based LLMs.
  • Performance Boost: Bayesian header fusion provided a clear benefit. For RobBERT in the direct five-way setting, macro F1 improved from 0.6680 to 0.6772.
  • Pipeline Success: Pipeline B (using OvR detectors) achieved the highest overall macro F1 of 0.6810 after Bayesian fusion.
  • Class Reliability: The "Decision" (beslissing) role is the most reliably detected when "None" sentences are excluded (F1 > 0.91), but its performance drops when the model has to filter through the entire document.
Table 2. Overall supervised model performance before and after Bayesian updating

Limitations and Future Work

  • Corpus Size: The study is limited to 100 decisions. While high-quality, larger datasets are needed to test generalization across more diverse legal sub-domains.
  • Context Blindness: The current models classify each sentence independently. Because rhetorical roles often shift in sequences, the authors suggest that context-aware architectures (like CRFs or long-context transformers) are a natural next step.
  • Intrinsic vs. Extrinsic: The evaluation was "intrinsic" (measuring F1/Accuracy). The authors recommend future work test these models in "extrinsic" deployments, such as actual legal search engines or summarization tools, to validate the suggested task-specific mappings.

Summary updated

Abstract

Classifying the material facts, submissions, judicial reasoning, and the court's decision (rhetorical roles) in case law is a prerequisite for high-quality performance across virtually all NLP-in-law tasks, including legal information retrieval, summarisation, and LLM-based analysis. These rhetorical roles often intertwine and shift sentence by sentence, making reliable classification difficult. This paper introduces the first sentence-level annotated dataset of Dutch judicial decisions, labelled across four legally meaningful categories: material facts, submissions, judicial reasoning, and the court's decision, alongside non-annotated (None) content, and compares several classification strategies, including a staged pipeline that separates None detection from the rhetorical roles, and a Bayesian update that leverages information about the section-header a sentence belongs to. Our results show that no single strategy dominates on both precision and recall: header priors help most when the None sentences are excluded and offer little benefit for None detection itself. These trade-offs matter in practice. For instance, extractive summarisation depends more on recall so it does not miss relevant facts, whereas outcome prediction depends more on precision regarding decision and facts sentences so that the predicted ruling is not contaminated by noise. We use these trade-offs and map the approaches tested in this study onto a set of representative legal NLP use cases.
BibTeX
@misc{sheth2026it,
  author = {Samyak Sheth and Gerasimos Spanakis and Gijs van Dijck},
  title = {{It Depends on the Task: Rethinking Rhetorical Segmentation of Dutch Judicial Decisions}},
  year = {2026},
  eprint = {2611.20868},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20868},
}
Figure