Law Retrieval with Supervised Contrastive Learning Using the Hierarchical Structure of Law

Jungmin Choi, Ukyo Honda, Taro Watanabe, Hiroki Ouchi, Kentaro InuiPACLIC 2022 · OpenReview

Introduction

Retrieving relevant legal articles is a significant challenge for non-experts due to the vocabulary mismatch problem. Legal statutes are often written in abstract, technical language that differs substantially from the natural language used by laypeople to describe their situations. While modern language models like BERT excel at semantic similarity, they often fail in "hard cases" where there is minimal keyword overlap between the query and the target article.

The researchers propose a novel approach that utilizes the inherent organizational logic of legal codes—such as Parts, Chapters, and Sections—to improve retrieval. Their method builds upon Sentence-BERT (SBERT) by introducing Supervised Contrastive Learning (SCL).

In a standard SBERT setup, the model is trained to minimize cosine similarity loss, aligning query and article embeddings. The authors' innovation adds a second loss function that constrains the geometry of the embedding space based on the law's hierarchical structure.

The model uses contrastive learning to pull articles from the same section closer together and push articles from different sections apart in the embedding space.

By forcing articles within the same section (e.g., "Possessory Rights") to cluster together, the model creates a "neighborhood" effect. If a query is semantically close to one "easy" article in a section, it effectively becomes close to all related articles in that section, including "hard" articles that might lack direct keyword overlap. This is referred to as the "Anchor Effect."

Experimental Setup

The model was evaluated using the COLIEE (Competition on Legal Information Extraction/Entailment) Task 3 dataset, which is based on the Japanese Civil Code. The task involves retrieving specific articles required to answer legal queries, often derived from bar exams. The training set consists of 725 queries, with 81 queries used for testing.

Key Results

The "Baseline + Contrastive" approach was compared against a fine-tuned BERT baseline and previous state-of-the-art methods. The results demonstrate that explicitly training the embedding space geometry via contrastive learning is more effective than simply appending section names to the text.

Comparison of retrieval performance (Recall@10) across different difficulty levels, highlighting the significant improvement in Hard-Anchored cases.

The most notable improvement occurred in "Hard-Anchored" cases—scenarios where the correct article is difficult to find individually but resides near an "easy" anchor article. In these cases, the proposed model achieved an R@10 of 78.33, a substantial jump from the baseline's 70.00. Overall performance also improved, reaching an R@10 of 88.48, while maintaining competitive precision.

Discussion and Limitations

The study highlights that the hierarchical structure of law is a powerful feature when encoded directly into the model's embedding space. This approach bridges the gap between layperson language and technical legal-ese, making legal systems more accessible to the public.

However, challenges remain. The model still struggles with:

  • Coreference Resolution: Handling complex pronouns and linguistic references within queries.
  • Hypernym Detection: Mapping concrete examples (e.g., "jewelry") to broad legal categories (e.g., "movables") when they are not conceptually close in pre-trained models.

Future research will explore integrating logic-based approaches or knowledge graphs to address these deeper semantic gaps.

Summary updated

BibTeX
@inproceedings{choi2022law,
  author = {Jungmin Choi and Ukyo Honda and Taro Watanabe and Hiroki Ouchi and Kentaro Inui},
  title = {{Law Retrieval with Supervised Contrastive Learning Using the Hierarchical Structure of Law}},
  booktitle = {PACLIC},
  year = {2022},
  url = {https://openreview.net/forum?id=kh7Cm6JbWGS},
}
Figure