Sentence Extraction for Legal Text Summarisation

Ben Hachey, Claire GroverIJCAI 2005

Abstract

This paper presents a system for the automatic summarisation of complex legal judgments from the UK House of Lords. By framing summarisation as a sentence extraction and ranking task, the authors evaluate Naïve Bayes (NB) and Maximum Entropy (ME) classifiers using a diverse set of linguistic and structural features. Results demonstrate that these models significantly outperform lead-based baselines, with NB achieving a peak F-score of 29.8 and a correlation coefficient of 0.235. The study emphasizes the importance of rhetorical roles and feature automation for creating portable, user-tailored legal summaries.

Introduction

Legal professionals and researchers frequently encounter the challenge of navigating massive volumes of judicial text. Unlike standard news articles, a legal judgment is a highly structured argument where the rhetorical role of a sentence—such as whether it represents a final decision or background argumentation—is as critical as its factual content. Existing summarisation tools often lack the flexibility to capture these nuances or rely too heavily on manual, domain-specific rules. This research aims to develop a "portable" system capable of producing summaries of varying lengths by ranking sentences according to their "abstract-worthiness."

Methodology

The researchers treat summarisation as a sentence extraction task, comparing two machine learning approaches: Naïve Bayes (NB) and Maximum Entropy (ME). The core of the system lies in its feature engineering, which avoids hand-crafted lexicons in favor of automatically computable attributes:

  • Structural Features: The position of a sentence within the document, speech, or paragraph.
  • Content Features: Standard $tf*idf$ weights for thematic words and sentence length.
  • Linguistic Features: Presence of quotation marks, named entities, and "cue phrases."

To ensure portability across legal domains, the authors replace traditional hand-coded phrase lists with syntactic features such as verb tense, aspect, and part-of-speech tags.

Dataset: The HOLJ Corpus

The authors introduced the UK House of Lords Judgments (HOLJ) corpus to train and test their models. The dataset consists of 69 judgments annotated for rhetorical roles. A subset of 47 judgments was further annotated for relevance by matching sentences to human-written summaries. This multi-layered corpus includes manual annotations for roles and relevance alongside automatic linguistic markup.

Key Results and Discussion

The experimental results indicate that both machine learning models significantly outperform the "lead-based" baseline, which achieved an F-score of 23.8. As shown in the performance metrics, the Naïve Bayes model reached a top F-score of 29.8 when incorporating sentence length features.

Performance metrics (Precision, Recall, F-score) for NB and ME models across incremental feature sets.

Beyond simple classification, the authors argue that summarisation is best framed as a ranking task. Using the point-biserial correlation coefficient ($r_{pb}$) to measure ranking quality, Naïve Bayes proved more robust than Maximum Entropy. While NB successfully incorporated all features to reach a correlation of 0.235, the ME model struggled with certain features, such as location, which resulted in negative correlation in some configurations.

Point-biserial correlation coefficients (I and C) for NB and ME models, illustrating ranking quality.

Conclusion

While limited by a relatively small dataset, this research demonstrates that automated feature engineering and rhetorical role categorisation can effectively identify relevant sentences in legal texts. The findings suggest that the "best" model depends heavily on the chosen metric; while NB performed better in overall F-score and correlation, ME showed high precision in specific contexts. Future work will involve user studies to determine if these statistical improvements align with human utility and exploring how rhetorical roles can be used to reorder sentences for more coherent final summaries.

Summary updated

BibTeX
@inproceedings{hachey2005sentence,
  author = {Ben Hachey and Claire Grover},
  title = {{Sentence Extraction for Legal Text Summarisation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2005},
}
Figure