Abstract
This paper investigates the use of fine-tuned BERT models to semi-automate the triage of free legal assistance requests. By categorizing layperson descriptions into specific legal areas, the proposed system aims to alleviate the intake bottleneck for non-profit legal services. We evaluate three annotation weighting strategies—majority-vote, confidence-weighted, and annotator-weighted—finding that the annotator-weighted model performs best with an F1-score of 0.671. However, a fairness audit reveals significant performance disparities across demographic groups, particularly for seniors and Aboriginal/Torres Strait Islander seekers.
Introduction
In Australia, millions of individuals face unmet legal needs while free legal services remain chronically under-resourced. A critical bottleneck is the "triage" process: identifying the specific legal issue from an informal, non-expert description. Help-seekers often use non-technical language (e.g., describing a "guardianship" issue as a "family problem"), and their communication styles vary by social and educational background. This research explores whether Natural Language Processing (NLP) can bridge this gap by automatically assigning "area-of-law" categories to real-world requests, enabling organizations like Justice Connect to route cases to pro bono lawyers more efficiently.
Methods and Data
The researchers fine-tuned a BERT (Bidirectional Encoder Representations from Transformers) model on a proprietary dataset from the Justice Connect online intake tool. The dataset consists of 4,062 unique, de-identified descriptions of legal problems tagged by 243 volunteer lawyers across 33 areas of law.
To address the inherent ambiguity in legal labeling, three annotation weighting strategies were implemented:
- Majority-vote: Labels were applied only if at least half of the annotators agreed.
- Confidence-weighted: Lawyer-annotators provided a "certainty" score (1–100) for each label, which was factored into the model training.
- Annotator-weighted: Labels were weighted by the proportion of annotators who assigned them (requiring at least 3 annotators per instance).
Additionally, the study explored hierarchical grouping, clustering the 33 categories into broader "themes" or "legal specializations" to determine if coarser granularity improved predictive accuracy.
Results
The BERT-based models significantly outperformed simple baselines. The Annotator-weighted BERT model emerged as the top performer, achieving a Precision of 0.781, a Recall of 0.588, and an F1-score of 0.671.
Key findings include:
- Precision Priority: In pro bono triage, precision is prioritized over recall. It is more efficient for the system to remain conservative, suggesting a category only when highly certain, to avoid the overhead of misassigning cases to unqualified lawyers.
- Performance Tiers: Legal areas were categorized into four tiers based on model success. Tier I issues (e.g., Migration, Employment) showed high F-scores, while Tier IV issues (e.g., IT, Environment) suffered from data sparsity.
- Grouping Trade-offs: While grouping labels into broader themes improved recall (up to 0.629), it did not enhance precision, suggesting that the loss of specific detail may not be beneficial for triage.
Discussion and Fairness Audit
A critical insight from this study is the model's inequitable performance across demographics. Despite having sufficient data, the model performed significantly worse for "Seniors" (F1: 0.543) and Aboriginal and Torres Strait Islander groups (F1: 0.638). The researchers hypothesize that these disparities stem from linguistic style differences rather than the legal issues themselves, as BERT may be less effective at processing language that deviates from its standard pre-training data.
Future work will focus on addressing these biases through adversarial learning or null space projection to create models invariant to demographic features. The team also plans to expand the study to include more diverse groups, such as individuals with disabilities or those from culturally and linguistically diverse (CALD) backgrounds.
Summary updated