The Law and NLP: Bridging Disciplinary Disconnects

Robert Mahari, Dominik Stammbach, Elliott Ash, Alex 'Sandy' PentlandarXiv 2023 · DOI · arXiv

Abstract

While Natural Language Processing (NLP) has the potential to revolutionize the legal field and address the global "access to justice" crisis, a significant disconnect exists between computational research and legal practice. This paper analyzes the current state of legal NLP, identifying a misalignment where researchers focus on technically interesting but practically flawed tasks like Legal Judgment Prediction (LJP), while neglecting the high-impact, "mundane" needs of the legal community. By bridging this gap through interdisciplinary collaboration and a focus on document-level analysis, the field can better serve the needs of both the legal profession and the public.

Introduction

The legal system is currently facing an "access to justice" crisis. In the United States, approximately 92% of civil legal problems for low-income individuals go unaddressed, and public defenders are systematically overwhelmed by massive caseloads. Given that the legal field is built entirely on language—statutes, precedents, and arguments—NLP technology appears to be a natural fit. However, adoption has been sluggish.

The authors identify a "disciplinary disconnect" as the primary culprit. NLP researchers often treat legal data as a standard text classification problem without understanding the nuances of legal practice. Conversely, the legal community remains largely unaware of the tools being developed in computer science labs. The result is a research ecosystem that produces models lawyers cannot use, while failing to provide the robust document analysis tools they desperately need.

Methodology

The authors employed a two-pronged approach to diagnose the state of the field:

  1. Rapid Literature Review: The researchers conducted "forward citation chasing" starting from a seminal 2020 survey paper. They manually annotated 171 papers published between 2020 and 2023 to determine their primary contribution (NLP, Legal, or Interdisciplinary) and their specific research objectives.
  2. Practitioner Survey: To ground their findings in reality, the authors analyzed a survey of over 50 senior attorneys from 17 countries (conducted via Harvard Law School’s Leadership in Law Firms program). Practitioners were asked to identify the legal tasks they would most like generative AI to simplify.

A primary finding of the literature review is the prevalence of Legal Judgment Prediction (LJP)—the task of predicting a court's decision based on the facts of a case. LJP is the most popular research topic, appearing in 20% of the reviewed papers. However, the authors argue this task is fundamentally flawed.

Judges typically write the "facts" of an opinion after they have decided the outcome, often framing the narrative to justify the conclusion. Using these post-hoc facts to "predict" the outcome is a form of data leakage that makes the task technically trivial and practically useless for a lawyer preparing for trial. Furthermore, while 20% of research focuses on LJP, only 3% of surveyed practitioners expressed interest in simulating or predicting legal outcomes.

Practitioner Needs: The "Boring" High-Impact Tools

While researchers chase LJP, lawyers are looking for help with the "boring" aspects of legal work. The survey of senior attorneys showed an overwhelming preference for document-level tasks:

  • Document Review and Summarization: 71% of surveyed attorneys (42/59 responses) identified this as their top priority. They need tools to manage the massive volume of discovery documents and to find "good law" (ensuring a case hasn't been overturned).
  • Administrative Support: 14/59 responses highlighted a need for help with billing, intake, and administrative workflows.
  • Legal Research: 11/59 responses focused on regulatory review and standard legal research.

The Echo Chamber Effect

The literature review revealed a concerning lack of interdisciplinary work. 90% of the 171 analyzed papers were produced and consumed solely by the NLP community. Only 10% were classified as truly interdisciplinary, and a staggering single law review article cited the seed paper used for the review. This suggests that legal NLP is currently "talking to itself," detached from the legal scholars and practitioners who understand the domain's requirements.

A Path Forward

To move the field toward meaningful impact, the authors advocate for several strategic shifts:

  • Retrieval Augmented Generation (RAG): Developing systems that can safely query a firm's internal, private knowledge base without leaking confidential data.
  • Persuasive Reasoning: Moving beyond simple classification to understand the rhetoric and pragmatics of legal arguments.
  • Access to Justice as a North Star: Shifting the shared normative goal of the field from simulating judges to empowering non-lawyers and overworked public defenders.
  • Closer Collaboration: Encouraging NLP researchers to co-author with legal scholars to ensure that the tasks being solved are legally meaningful and ethically sound.

Conclusion

The disconnect between NLP research and legal practice represents a missed opportunity to address critical societal needs. By moving away from flawed tasks like LJP and focusing on the practical, document-level challenges faced by attorneys, the NLP community can create tools that not only improve legal efficiency but also broaden access to justice for those who need it most.

Summary updated

Abstract

Legal practice is intrinsically rooted in the fabric of language, yet legal practitioners and scholars have been slow to adopt tools from natural language processing (NLP). At the same time, the legal system is experiencing an access to justice crisis, which could be partially alleviated with NLP. In this position paper, we argue that the slow uptake of NLP in legal practice is exacerbated by a disconnect between the needs of the legal community and the focus of NLP researchers. In a review of recent trends in the legal NLP literature, we find limited overlap between the legal NLP community and legal academia. Our interpretation is that some of the most popular legal NLP tasks fail to address the needs of legal practitioners. We discuss examples of legal NLP tasks that promise to bridge disciplinary disconnects and highlight interesting areas for legal NLP research that remain underexplored.
BibTeX
@misc{mahari2023law,
  author = {Robert Mahari and Dominik Stammbach and Elliott Ash and Alex 'Sandy' Pentland},
  title = {{The Law and NLP: Bridging Disciplinary Disconnects}},
  year = {2023},
  eprint = {2310.14346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.14346},
}
Figure