TL;DR
Researchers have developed an AI assistant based on the TRACE framework to automate the anti-corruption review of draft legislation. By decomposing the task into a structured pipeline—combining deterministic parsing with LLM-guided reasoning—the system allows mid-sized local models to achieve performance comparable to top-tier cloud models in identifying regulatory defects.
What Problem Does This Paper Address?
In many jurisdictions, including Ukraine, draft laws must undergo "anti-corruption expertise" to identify provisions that might facilitate graft. This is a labor-intensive process performed by agencies like Ukraine’s National Agency on Corruption Prevention (NAZK). The sheer volume of incoming legislation far outpaces the capacity of human experts to review every bill thoroughly.
While Large Language Models (LLMs) show promise in legal tech, existing benchmarks focus on general tasks like bar exams or contract classification. This paper tackles the specific challenge of provision-by-provision reproduction of expert findings: can an AI pinpoint the exact article an expert would criticize and correctly categorize the specific corruption risk?
Methods
The authors implement the TRACE architecture, which emphasizes "computational parsimony"—using deterministic code for what it does best and reserving LLMs for high-level reasoning. The pipeline consists of four layers:
- L1 (Deterministic Parsing): Segments the bill into structural units (articles, paragraphs) and normalizes addresses. For amending bills, it maps changes back to the original statute.
- L2a (Retrieval): Uses BM25 to select candidate provisions and retrieve relevant historical NAZK precedents.
- L2b (Qualification): An LLM (like Gemma 4 31B or Claude Sonnet 5.5) qualifies one provision at a time using a structured reasoning schema. Instead of picking a risk factor directly, the model identifies the addressee, missing safeguards, and risk types.
- L3 (Orchestration & Ranking): Deterministic rules map the LLM's output to the official 30-factor NAZK taxonomy and rank the resulting "flags" by severity.
This approach is contrasted with a "monolithic" baseline, where the entire bill and methodology are dumped into a single long-context prompt.
Datasets
The authors introduced a new open benchmark derived from real-world data:
- NAZK Benchmark: 132 draft-to-conclusion pairs containing 474 model-extracted findings.
- Splits: The data is partitioned into
dev,test,holdout, and a largertest100set to ensure rigorous evaluation, including "blind" tests where the system was not tuned on the specific acts.
Key Insights and Ideas
- Structure Trumps Scale: For local models like Gemma 31B, the structured pipeline significantly outperformed the monolithic prompt. The pipeline "sharpens" the model's focus, preventing it from giving generic answers.
- The "Qualification Gap": The biggest bottleneck isn't finding the problematic provision; it's correctly identifying the type of risk. Models often flagged the right article but assigned a different regulatory defect than the human experts.
- Contextual Blindness: Current models struggle with "legal collisions"—provisions that are only corrupt because they conflict with a different law not present in the draft. Since the LLM looks at provisions in isolation, these risks go undetected.
- LLM-as-a-Judge Reliability: The study used
gpt-6.1-solas a judge to compare system flags against NAZK findings. They found high inter-route consistency (98.6% agreement), suggesting that LLM judges are stable enough for this type of comparative architectural research.
Key Results
- Recall: On the
test100set, the pipeline matched 28% of NAZK findings within its top ten flags using the local Gemma model, and 30% using Claude Sonnet 5.5. When considering all flags, recall reached 43-46%. - Efficiency: For cloud models like Sonnet, the structured pipeline reduced the number of "false alarm" flags nearly fourfold compared to the monolithic prompt while maintaining similar recall.
- Local vs. Cloud: With the addition of an L3 prioritization layer, the gap between the 31B local model and the much larger cloud model narrowed to within the margin of sampling noise.
Limitations and Future Work
The authors are transparent about several hurdles:
- Legal Collisions: The system is currently "blind" to external statutory context. Future work involves integrating a "legislation knowledge base" to detect conflicts between laws.
- Human Validation: While the LLM-judge is consistent, it hasn't been fully calibrated against human anti-corruption specialists. The current scores reflect agreement with model-extracted findings, not necessarily absolute ground truth.
- The "Corruption-Risk Dial": The authors propose a visual interface to provide temporal context (e.g., how often an article has been amended), which could help human experts catch the nuances the AI misses.
As shown in the failure analysis above, "location found, risk different" remains the dominant failure mode, highlighting that the next frontier is improving the depth of legal reasoning within the qualification stage.
Summary updated