Legal Area Classification: A Comparative Study of Text Classifiers on Singapore Supreme Court Judgments

Jerrold Soh, How Khang Lim, Ian Ernst ChaiNAACL 2019

TL;DR

This paper compares traditional and state-of-the-art machine learning models for classifying legal judgments into specific areas of law. They found that while advanced models show promise, traditional methods like Latent Semantic Analysis (LSA) surprisingly held their own, highlighting the unique challenges and opportunities in applying NLP to lengthy, specialized legal texts.

What Problem Does This Paper Address?

Classifying legal documents into distinct "legal areas" (like contract law, tort law, etc.) is crucial for efficient legal research, case triage, and organizing vast archives of legal precedent. However, this task, termed "legal area classification," is relatively underexplored. The authors identify two main challenges: 1) the scarcity of labeled legal documents, especially in smaller jurisdictions like Singapore, and 2) the inherent length and complexity of legal texts, which differ significantly from the shorter documents (like tweets or reviews) typically used to benchmark NLP models. This paper aims to bridge this gap by comparing various classification approaches on a novel dataset of Singapore Supreme Court judgments.

Methods

The researchers evaluated four categories of text classification models:

  • Baseline Models:
    • basepdf: A simple dummy classifier predicting the most frequent label.
  • countm: A keyword-matching strategy that emulates how lawyers might search for terms related to a legal area.
  • Topic Models:
    • lsak: Latent Semantic Analysis (LSA) combined with linear Support Vector Machines (linSVMs). LSA was chosen as a well-established benchmark in legal NLP.
  • Word Embedding Feature Models:
    • These models leverage pre-trained word vectors (GloVe) to represent judgments. They explored three pooling strategies: average pooling (gloveavg), max pooling (glovemax), and a shallow Convolutional Neural Network (glovecnn).
  • Pre-trained Language Models:
    • bertlarge and bertbase: State-of-the-art transformer models (BERT) fine-tuned for the task.
  • ulm fit: A model based on a stacked-LSTM architecture, known for handling longer sequences.

The study also meticulously addressed data imbalance by using stratified sampling and weighted loss functions.

Datasets

The primary dataset used is a novel collection of 6,227 Singapore Supreme Court judgments issued between January 2000 and February 2019. These judgments were originally in PDF format and were parsed to extract text and labels. The raw dataset contained 51 legal area labels, which were then manually merged and truncated to the 30 most frequent areas, with the rest mapped to an "others" category. This dataset is notable for its size (over 6,000 documents) and the length of individual documents (median of 6,968 tokens), making it a challenging benchmark for NLP models.

Key Insights and Ideas

  • Traditional Models Still Relevant: Surprisingly, LSA-based models performed very competitively, even outperforming some advanced deep learning models on certain metrics, especially at higher data availability. This suggests that the "bag-of-words" approach, when combined with dimensionality reduction like LSA, can still capture significant legal information.
  • Length Matters: The inherent length of legal documents poses a challenge for models like BERT, which have a fixed input token limit (512 tokens). The authors had to truncate judgments, potentially losing valuable context, which might explain why BERT didn't universally dominate.
  • Word Embeddings Show Promise: Models using pre-trained word embeddings (GloVe) with appropriate pooling strategies (especially glovecnn) demonstrated strong performance, indicating that semantic representations are effective for legal text.
  • Recall vs. Precision Trade-off: While LSA models excelled in precision (correctly identifying labels), they struggled with recall (finding all relevant labels). Keyword-based approaches, surprisingly, showed better recall, highlighting their utility in ensuring no relevant precedents are missed, a critical aspect in legal practice.
  • Data Scarcity is a Hurdle, but Not Insurmountable: Even with limited data (10% subset), most ML approaches outperformed the simple baselines, suggesting that deep learning models can learn effectively with relatively few labeled examples, especially when pre-trained.

Key Results

  • Overall Performance: All ML models consistently outperformed the baseline basepdf and countm models across various metrics (F1 score, precision, recall).
  • LSA's Strength: lsa250 emerged as the best performing model on the 100% data subset for both micro- and macro-averaged F1 scores, driven by high precision.
  • BERT and ULMFIT: These advanced models performed well, particularly on smaller training subsets, with bertlarge and bertbase achieving strong macro-F1 scores. ulm fit also showed good performance, especially on the 10% subset.
  • Word Embedding Models: glovecnn and gloveavg achieved competitive F1 scores, with glovecnn outperforming gloveavg and glovemax.
  • Recall: BERT models and glovecnn showed the best micro- and macro-recall on the 10% and 50% subsets, while glovecnn led for the 100% subset.

Limitations and Future Work

  • BERT Input Length: The fixed input size of BERT models limited their ability to process entire judgments, forcing truncation. The authors suggest this is an area for future experimentation.
  • Out-of-the-Box Performance: The study focused on comparing models "out-of-the-box" without extensive hyperparameter tuning or cross-validation. Further optimization could yield different results.
  • Future Directions:
    • Developing law-specific datasets and baselines for training and benchmarking legal text classifiers.
  • Exploring representation learning techniques that can scale to long texts, such as advanced CNN architectures, contextualized word embeddings, and feature extraction methods for models like BERT.
  • Adapting models to better handle the unique characteristics of legal corpora.

Summary updated

BibTeX
@inproceedings{soh2019legal,
  author = {Jerrold Soh and How Khang Lim and Ian Ernst Chai},
  title = {{Legal Area Classification: A Comparative Study of Text Classifiers on Singapore Supreme Court Judgments}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure