Legal Linking: Citation Resolution and Suggestion in Constitutional Law

Robert Shaffer, Stephen MayhewNAACL 2019

TL;DR

This paper introduces a new task and dataset for automatically linking paragraphs from US Supreme Court cases to specific clauses or amendments in the US Constitution. They developed a neural network model that outperforms rule-based and linear baselines, particularly for identifying indirect or abstract legal connections, suggesting its utility for legal research and discovering novel legal arguments.

What Problem Does This Paper Address?

Legal professionals and researchers often need to understand how specific legal documents, like court opinions, relate to foundational legal texts, such as the US Constitution. Manually identifying these connections is time-consuming and labor-intensive, especially when dealing with a large corpus of documents. This is crucial for tasks like understanding how different constitutional rights are invoked together in legal arguments or for analyzing the frequency of constitutional clause usage in judicial opinions. Existing systems are often too coarse or not designed for inferring links from plain text excerpts.

Methods

The authors tackle this problem by framing it as a binary classification task: does a given paragraph from a court case link to a specific constitutional provision? They explore three main approaches:

  • Rule-based system: This baseline uses manually defined rules to map amendment names and common phrases to specific constitutional provisions.
  • Linear Model: A multi-label classifier using logistic regression with unigrams, bigrams, and trigrams as features. To handle imbalanced data, they subsampled unmatched examples and downweighted negative examples.
  • Neural Network Model: This is their primary contribution. It uses a BERT-based token embedding layer for input text and a bidirectional LSTM for document embedding. The model learns a similarity measure between paragraph and constitutional provision embeddings. A key innovation is their "stripping" technique, where they remove explicit rule strings (like "First Amendment") from a portion of the training data. This forces the model to learn more abstract connections rather than just memorizing explicit mentions.

Datasets

  • Training Data: The authors scraped text from the Cornell Legal Information Institute's repository of US Supreme Court opinions. Initially, they identified paragraphs with hyperlinks to the US Constitution, resulting in approximately 328k unique paragraphs and 11k links. This data was then processed to create a "stripped" version by removing explicit rule strings from half of the examples.
  • Evaluation Data: A manually annotated set of 1241 paragraphs from five specific Supreme Court cases (Griswold v. Connecticut, Miranda v. Arizona, US v. Nixon, Texas v. Johnson, and NFIB v. Sebelius). These cases were chosen for their relevance to constitutional law questions and varying rhetorical styles, making them suitable for evaluating the model's ability to handle both direct and indirect links.

Key Insights and Ideas

  • The Power of "Stripping": Removing explicit rule strings from training data significantly improves the neural network's ability to generalize and identify less obvious links. This suggests that models can easily overfit to superficial textual cues if not properly constrained.
  • Abstract vs. Direct Links: The performance gap between the neural network and rule-based systems is most pronounced for abstract or indirect references. This highlights the neural network's advantage in capturing nuanced legal reasoning that goes beyond keyword matching.
  • Data Quality Matters: The initial distantly-supervised data, while useful for bootstrapping, is inconsistent and incomplete. Manual annotation, even for a subset, is crucial for reliable evaluation and understanding model performance.
  • Legal Language Nuance: The example of the Eighth Amendment and "cruel," "punishment," and "unusual" demonstrates how neural models can learn to associate concepts even when explicit keywords are absent or reordered, a feat difficult for rule-based systems.

Key Results

  • The rule-based system achieved high precision (91.8%) but low recall (47.0%) on the manually annotated test set.
  • The linear and neural network models trained on the original (unstripped) data showed similar recall to the rule-based system but lower precision.
  • The neural network model trained on the "stripped" data achieved the highest recall (56.2%) and the best overall F1 score (64.8%), outperforming all other approaches. This demonstrates the effectiveness of the stripping technique.

Limitations and Future Work

  • Data Incompleteness: The authors acknowledge that their distantly-supervised training data is incomplete. They suggest that future work could focus on more sophisticated automatic or distant labeling techniques.
  • Problem Modeling: While they present a binary classification approach, they note that further exploration into problem modeling could yield improvements.
  • Discovering Novel Links: The success of the neural network on indirect links suggests its potential for discovering new, previously unarticulated connections between legal texts, which could be a significant area for future research.
  • Broader Applications: The authors envision their work being useful for political science research, such as studying litigation trends or rhetorical co-citation of constitutional sections.

Summary updated

BibTeX
@inproceedings{shaffer2019legal,
  author = {Robert Shaffer and Stephen Mayhew},
  title = {{Legal Linking: Citation Resolution and Suggestion in Constitutional Law}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure