Searching for Legal Documents at Paragraph Level: Automating Label Generation and Use of an Extended Attention Mask for Boosting Neural Models of Semantic Similarity

Li Tang, Simon ClematideNLLP @ EMNLP 2021 · OpenReview

Introduction

Legal Information Retrieval (IR) is a complex task due to the length and specialized language of legal documents. Finding relevant past court decisions often requires mastering intricate Boolean queries, which can be inefficient for both experts and laypeople. While Transformer-based models like BERT have transformed general Natural Language Processing (NLP), they face significant hurdles in the legal domain: a lack of large-scale labeled datasets, a substantial domain gap from general-purpose training data, and the prevalence of "noise" in the form of boilerplate legal language. This paper addresses these challenges by introducing an automated method for generating semantic similarity labels and a novel Extended Attention Mask to enhance neural model performance on paragraph-level search.

Methodology

The authors propose two key innovations to improve semantic similarity modeling in legal texts.

Automated Label Generation

To bypass the high cost of expert annotation, the researchers employed "bibliographic coupling." In this framework, two case paragraphs are deemed semantically similar if they cite at least two of the same specific statute articles. This citation-based approach provides a high-quality, automated proxy for relevance, enabling the creation of large-scale evaluation datasets without manual labeling.

Extended Attention Mask

Standard Transformer models use attention masks to ignore padding tokens. The authors introduced an Extended Attention Mask that incorporates a focus mask based on Inverse Document Frequency (IDF) scores. By identifying and masking out tokens with low IDF scores—representing common, uninformative words—the model's self-attention mechanism is forced to focus exclusively on high-information legal terms.

The Extended Attention Mask is generated by combining a standard padding mask with a TF-IDF-based focus mask to prioritize informative tokens.

Experimental Setup

The study focused on German-language legal documents from Switzerland, utilizing a dataset of 7,562 case law paragraphs and 38,994 statute paragraphs. The automated labeling process identified 15,348 case pairs to serve as ground truth. The researchers evaluated several models, including a GermanBERT variant pre-trained on a corpus containing 20% legal text, using Mean Average Precision (AP) as the primary metric.

Results and Discussion

The experimental results demonstrate that the Extended Attention Mask provides a consistent performance boost across BERT-based architectures. The GermanBERT model with the extended mask achieved a Mean AP of 48.19%, significantly outperforming the standard GermanBERT (44.38%) and a non-neural TF-IDF baseline (41.30%).

Model evaluation results across varying IDF thresholds. The zoomed-in view shows a clear performance peak for neural models when the optimal amount of noise is filtered.

The analysis showed that performance peaks at an IDF threshold of approximately 2.0. This suggests that explicitly quenching the signal from low-information words directly within the attention layer allows the model to better capture the core legal concepts. Furthermore, the success of the citation-based labeling method confirms that citations serve as powerful semantic anchors in the legal domain.

Conclusion

This work demonstrates that domain-specific architectural adjustments, such as the Extended Attention Mask, can significantly improve the retrieval of complex legal documents. By leveraging citation networks for automated labeling, researchers can overcome data scarcity and build more effective tools for legal professionals. Future research will explore fine-tuning models directly on these automated labels and extending the approach to other languages and legal jurisdictions.

Summary updated

BibTeX
@inproceedings{tang2021searching,
  author = {Li Tang and Simon Clematide},
  title = {{Searching for Legal Documents at Paragraph Level: Automating Label Generation and Use of an Extended Attention Mask for Boosting Neural Models of Semantic Similarity}},
  booktitle = {NLLP @ EMNLP},
  year = {2021},
  url = {https://openreview.net/forum?id=soSguEBaOg},
}
Figure