Learning Policy Levers: Toward Automated Policy Analysis Using Judicial Corpora

Elliott Ash, Daniel L. Chen, Raúl Delgado, Eduardo Fierro, Shasha LinToulouse 2018

Abstract

To build inputs for end-to-end machine learning estimates of the causal impacts of law, we consider the problem of automatically classifying cases by their policy impact. We propose and implement a semi-supervised multi-class learning model, with the training set being a hand-coded dataset of thousands of cases in over 20 politically salient policy topics. Using opinion text features as a set of predictors, our model can classify labeled cases by topic correctly 91% of the time. We then take the model to the broader set of unlabeled cases and show that it can identify new groups of cases by shared policy impact.

Introduction

In legal scholarship, understanding the causal impact of the law—how a specific ruling affects housing prices, birth outcomes, or religious sentiment—requires researchers to first categorize thousands of judicial opinions. Traditionally, this "coding" is done by hand, which is agonizingly slow and limits the scope of research to small, manageable samples. This paper introduces a semi-supervised machine learning framework designed to automatically classify judicial decisions by their policy impact, bridging the gap between massive judicial corpora and actionable policy analysis.

Methodology: Beyond Keyword Matching

The core of the proposed approach is a two-stage learning model that moves beyond simple keyword matching to leverage high-dimensional document embeddings.

  1. Feature Extraction: Using Paragraph Vector - Distributed Bag of Words (PV-DBOW), the model maps entire documents into a vector space. Unlike traditional "Bag of Words" models, PV-DBOW captures semantic meaning and context, allowing the model to understand complex legal relationships.
  2. Supervised Classification: A Multi-class Support Vector Machine (SVM) is trained on a labeled dataset (the Chicago Project) to recognize major policy categories.
  3. Unsupervised Discovery: For cases that do not fit known categories, the model employs Gaussian Mixture Models (GMM) to discover new clusters, effectively identifying emerging legal trends and "unlabeled" policy levers.
The model's document embeddings reveal a structured semantic landscape where related legal issues naturally cluster together in the vector space.

Key Results and Performance

The PV-DBOW + SVM model achieved a remarkable 91% accuracy in classifying labeled cases. This performance significantly outstrips traditional Bag-of-Words models (83.9%), proving that capturing the context and semantic structure of judicial language is vital for accurate policy classification.

Confusion matrix demonstrating the high precision of the classification model across various legal topics, supporting the 91% accuracy claim.

Beyond supervised accuracy, the model demonstrated powerful discovery capabilities:

  • Cluster Recovery: It correctly recovered 15 out of 22 human-defined policy topics in unsupervised tests without prior labeling.
  • Discovery at Scale: When applied to an external 17,000-case evaluation set, the model identified 33 new clusters, successfully mapping unlabeled cases to complex categories like "Piercing the Corporate Veil" (PCV) and the "Americans with Disabilities Act" (ADA) with high consistency.

Conclusion

By automating the identification of "policy levers" within judicial text, this framework enables researchers to analyze the downstream societal impacts of decisions at an unprecedented scale. While currently optimized for U.S. Federal Courts, the success of the embedding-based approach suggests potential for generalizability to other legal systems, eventually linking judicial outcomes to external economic and social datasets.

Summary updated

Abstract

To build inputs for end-to-end machine learning estimates of the causal impacts of law, we consider the problem of automatically classifying cases by their policy impact. We propose and implement a semi-supervised multi-class learning model, with the training set being a hand-coded dataset of thousands of cases in over 20 politically salient policy topics. Using opinion text features as a set of predictors, our model can classify labeled cases by topic correctly 91% of the time. We then take the model to the broader set of unlabeled cases and show that it can identify new groups of cases by shared policy impact.
BibTeX
@inproceedings{ash2018learning,
  author = {Elliott Ash and Daniel L. Chen and Raúl Delgado and Eduardo Fierro and Shasha Lin},
  title = {{Learning Policy Levers: Toward Automated Policy Analysis Using Judicial Corpora}},
  booktitle = {Toulouse},
  year = {2018},
}
Figure