Abstract
To build inputs for end-to-end machine learning estimates of the causal impacts of law, we consider the problem of automatically classifying cases by their policy impact. We propose and implement a semi-supervised multi-class learning model, with the training set being a hand-coded dataset of thousands of cases in over 20 politically salient policy topics. Using opinion text features as a set of predictors, our model can classify labeled cases by topic correctly 91% of the time. We then take the model to the broader set of unlabeled cases and show that it can identify new groups of cases by shared policy impact.
Introduction
In legal scholarship, understanding the causal impact of the law—how a specific ruling affects housing prices, birth outcomes, or religious sentiment—requires researchers to first categorize thousands of judicial opinions. Traditionally, this "coding" is done by hand, which is agonizingly slow and limits the scope of research to small, manageable samples. This paper introduces a semi-supervised machine learning framework designed to automatically classify judicial decisions by their policy impact, bridging the gap between massive judicial corpora and actionable policy analysis.
Methodology: Beyond Keyword Matching
The core of the proposed approach is a two-stage learning model that moves beyond simple keyword matching to leverage high-dimensional document embeddings.
- Feature Extraction: Using Paragraph Vector - Distributed Bag of Words (PV-DBOW), the model maps entire documents into a vector space. Unlike traditional "Bag of Words" models, PV-DBOW captures semantic meaning and context, allowing the model to understand complex legal relationships.
- Supervised Classification: A Multi-class Support Vector Machine (SVM) is trained on a labeled dataset (the Chicago Project) to recognize major policy categories.
- Unsupervised Discovery: For cases that do not fit known categories, the model employs Gaussian Mixture Models (GMM) to discover new clusters, effectively identifying emerging legal trends and "unlabeled" policy levers.
Key Results and Performance
The PV-DBOW + SVM model achieved a remarkable 91% accuracy in classifying labeled cases. This performance significantly outstrips traditional Bag-of-Words models (83.9%), proving that capturing the context and semantic structure of judicial language is vital for accurate policy classification.
Beyond supervised accuracy, the model demonstrated powerful discovery capabilities:
- Cluster Recovery: It correctly recovered 15 out of 22 human-defined policy topics in unsupervised tests without prior labeling.
- Discovery at Scale: When applied to an external 17,000-case evaluation set, the model identified 33 new clusters, successfully mapping unlabeled cases to complex categories like "Piercing the Corporate Veil" (PCV) and the "Americans with Disabilities Act" (ADA) with high consistency.
Conclusion
By automating the identification of "policy levers" within judicial text, this framework enables researchers to analyze the downstream societal impacts of decisions at an unprecedented scale. While currently optimized for U.S. Federal Courts, the success of the embedding-based approach suggests potential for generalizability to other legal systems, eventually linking judicial outcomes to external economic and social datasets.
Summary updated