TL;DR
This paper introduces the first sentence-level annotated dataset for rhetorical role classification in Dutch judicial decisions. The authors demonstrate that there is no "one-size-fits-all" model; instead, the optimal classification strategy—whether a direct multi-class approach or a staged pipeline—depends heavily on whether a downstream task prioritizes precision (like outcome prediction) or recall (like extractive summarization).
What Problem Does This Paper Address?
In the legal domain, knowing the "rhetorical role" of a sentence—whether it describes a fact, a party's submission, the court's reasoning, or the final ruling—is essential for high-performance NLP. However, judicial text is messy. Unlike a scientific paper with clear "Methods" or "Results" sections, Dutch judicial decisions often interleave these roles within a single passage. A sentence's position or section header isn't always a reliable indicator of its function.
The authors tackle two main gaps:
- Data Scarcity: There was previously no publicly available, sentence-level annotated corpus for Dutch case law.
- Strategy Evaluation: Most existing work treats role classification as a standard multi-class problem. This paper investigates whether specialized pipelines or Bayesian updates (using section headers as priors) can better handle the nuances of legal text.
Methods
The researchers evaluated several model families, including a lexical baseline (Logistic Regression with TF-IDF), Dutch encoder-based models (RobBERT and BERTje), and several instruction-tuned LLMs (Gemma, Qwen) in zero/few-shot settings.
They compared four classification formulations:
- Five-way Classification: A single model predicting all roles plus a "None" category.
- Pipeline A: A binary "None Detector" followed by a four-way role classifier.
- Pipeline B: A "None Detector" followed by independent One-vs-Rest (OvR) detectors for each role.
- Bayesian Updating: A novel post-processing step that adjusts the model's text-based probability using a "header prior" (how likely a role is to appear under a specific section header).
Datasets
The authors created a new experimental corpus of 100 Dutch judicial decisions (2015–2021) covering civil, criminal, and administrative law.
- Size: 8,391 sentences.
- Labels: materiele feiten (facts), proceshandelingen (submissions), beoordeling (reasoning), beslissing (decision), and None.
- Quality: High inter-annotator agreement (mean Cohen’s $\kappa = 0.871$).
Key Insights and Ideas
- "None" is the Hardest Part: Detecting whether a sentence has any legal role (None Detection) is significantly more difficult than distinguishing between the roles themselves. Neither Bayesian priors nor confidence thresholding significantly improved this stage.
- The Precision-Recall Trade-off: Different downstream tasks have different needs. For example, extractive summarization needs high recall to ensure no facts are missed. Conversely, outcome prediction needs high precision to ensure the model isn't "hallucinating" a ruling based on noisy, non-decision sentences.
- Headers are Signals, Not Rules: While section headers provide a strong hint, they are not definitive. The Bayesian update treats them as a probabilistic signal, which improved performance for role classification but proved less useful for filtering out "None" sentences.
- Pipelines Expose Trade-offs: While a staged pipeline (Pipeline B) only modestly outperformed a direct five-way classifier in aggregate F1, its real value lies in modularity. It allows developers to tune specific components (e.g., a high-precision "Decision" detector) without retraining the entire system.
Key Results
- Top Model: RobBERT consistently outperformed BERTje, lexical baselines, and prompting-based LLMs.
- Performance Boost: Bayesian header fusion provided a clear benefit. For RobBERT in the direct five-way setting, macro F1 improved from 0.6680 to 0.6772.
- Pipeline Success: Pipeline B (using OvR detectors) achieved the highest overall macro F1 of 0.6810 after Bayesian fusion.
- Class Reliability: The "Decision" (beslissing) role is the most reliably detected when "None" sentences are excluded (F1 > 0.91), but its performance drops when the model has to filter through the entire document.
Limitations and Future Work
- Corpus Size: The study is limited to 100 decisions. While high-quality, larger datasets are needed to test generalization across more diverse legal sub-domains.
- Context Blindness: The current models classify each sentence independently. Because rhetorical roles often shift in sequences, the authors suggest that context-aware architectures (like CRFs or long-context transformers) are a natural next step.
- Intrinsic vs. Extrinsic: The evaluation was "intrinsic" (measuring F1/Accuracy). The authors recommend future work test these models in "extrinsic" deployments, such as actual legal search engines or summarization tools, to validate the suggested task-specific mappings.
Summary updated