An AI Assistant for Automating Anti-Corruption Expertise of Draft Legislation Based on the TRACE Framework

Serhii Zabolotnii, Alina Tarasova

TL;DR

Researchers have developed an AI assistant based on the TRACE framework to automate the anti-corruption review of draft legislation. By decomposing the task into a structured pipeline—combining deterministic parsing with LLM-guided reasoning—the system allows mid-sized local models to achieve performance comparable to top-tier cloud models in identifying regulatory defects.

What Problem Does This Paper Address?

In many jurisdictions, including Ukraine, draft laws must undergo "anti-corruption expertise" to identify provisions that might facilitate graft. This is a labor-intensive process performed by agencies like Ukraine’s National Agency on Corruption Prevention (NAZK). The sheer volume of incoming legislation far outpaces the capacity of human experts to review every bill thoroughly.

While Large Language Models (LLMs) show promise in legal tech, existing benchmarks focus on general tasks like bar exams or contract classification. This paper tackles the specific challenge of provision-by-provision reproduction of expert findings: can an AI pinpoint the exact article an expert would criticize and correctly categorize the specific corruption risk?

Methods

The authors implement the TRACE architecture, which emphasizes "computational parsimony"—using deterministic code for what it does best and reserving LLMs for high-level reasoning. The pipeline consists of four layers:

  1. L1 (Deterministic Parsing): Segments the bill into structural units (articles, paragraphs) and normalizes addresses. For amending bills, it maps changes back to the original statute.
  2. L2a (Retrieval): Uses BM25 to select candidate provisions and retrieve relevant historical NAZK precedents.
  3. L2b (Qualification): An LLM (like Gemma 4 31B or Claude Sonnet 5.5) qualifies one provision at a time using a structured reasoning schema. Instead of picking a risk factor directly, the model identifies the addressee, missing safeguards, and risk types.
  4. L3 (Orchestration & Ranking): Deterministic rules map the LLM's output to the official 30-factor NAZK taxonomy and rank the resulting "flags" by severity.
The TRACE-based pipeline architecture.

This approach is contrasted with a "monolithic" baseline, where the entire bill and methodology are dumped into a single long-context prompt.

Datasets

The authors introduced a new open benchmark derived from real-world data:

  • NAZK Benchmark: 132 draft-to-conclusion pairs containing 474 model-extracted findings.
  • Splits: The data is partitioned into dev, test, holdout, and a larger test100 set to ensure rigorous evaluation, including "blind" tests where the system was not tuned on the specific acts.
Composition of the benchmark across various splits.

Key Insights and Ideas

  • Structure Trumps Scale: For local models like Gemma 31B, the structured pipeline significantly outperformed the monolithic prompt. The pipeline "sharpens" the model's focus, preventing it from giving generic answers.
  • The "Qualification Gap": The biggest bottleneck isn't finding the problematic provision; it's correctly identifying the type of risk. Models often flagged the right article but assigned a different regulatory defect than the human experts.
  • Contextual Blindness: Current models struggle with "legal collisions"—provisions that are only corrupt because they conflict with a different law not present in the draft. Since the LLM looks at provisions in isolation, these risks go undetected.
  • LLM-as-a-Judge Reliability: The study used gpt-6.1-sol as a judge to compare system flags against NAZK findings. They found high inter-route consistency (98.6% agreement), suggesting that LLM judges are stable enough for this type of comparative architectural research.

Key Results

  • Recall: On the test100 set, the pipeline matched 28% of NAZK findings within its top ten flags using the local Gemma model, and 30% using Claude Sonnet 5.5. When considering all flags, recall reached 43-46%.
  • Efficiency: For cloud models like Sonnet, the structured pipeline reduced the number of "false alarm" flags nearly fourfold compared to the monolithic prompt while maintaining similar recall.
  • Local vs. Cloud: With the addition of an L3 prioritization layer, the gap between the 31B local model and the much larger cloud model narrowed to within the margin of sampling noise.

Limitations and Future Work

The authors are transparent about several hurdles:

  1. Legal Collisions: The system is currently "blind" to external statutory context. Future work involves integrating a "legislation knowledge base" to detect conflicts between laws.
  2. Human Validation: While the LLM-judge is consistent, it hasn't been fully calibrated against human anti-corruption specialists. The current scores reflect agreement with model-extracted findings, not necessarily absolute ground truth.
  3. The "Corruption-Risk Dial": The authors propose a visual interface to provide temporal context (e.g., how often an article has been amended), which could help human experts catch the nuances the AI misses.
Analysis of where the pipeline fails to match expert findings.

As shown in the failure analysis above, "location found, risk different" remains the dominant failure mode, highlighting that the next frontier is improving the depth of legal reasoning within the qualification stage.

Summary updated

Abstract

We present an AI assistant for automating anti-corruption expertise of draft legislation based on the TRACE architecture, designed to screen bills for corruption risks: pinpoint provisions an expert would criticise and name the regulatory defect in each. The pipeline combines a deterministic parsing core, model-guided single-provision qualification, rule-based factor assignment, and external human oversight. To evaluate the assistant, we constructed an open benchmark from the published conclusions of Ukraine's National Agency on Corruption Prevention (NAZK), comprising 132 draft laws with 474 model-extracted findings. A system flag agrees with a finding when it targets a compatible provision and an LLM judge rates the risk as the same (strict) or partly the same (lenient). On 100 draft laws, the pipeline matches 0.28 of the findings within its top ten flags using the local Gemma 4 31B model, 0.30 with Sonnet 5.5, and 0.43-0.46 across all flags (non-blind; the frozen blind baseline achieves 0.21, 95% CI 0.16-0.26). Across all runs, most misses occur on provisions that the system located but qualified as a different risk, and none of the 32 legal-collision findings was matched. On 21 acts held out from tuning, the pipeline outperforms a monolithic prompt containing the full methodology under the strict measure, though not under the lenient one, and Opus 5.5 adds nothing over Sonnet. The reported scores reflect agreement with model-extracted findings rated by an LLM judge whose two access routes agree on 73 of 74 labels, without human expert validation.
BibTeX
@misc{zabolotnii2026ai,
  author = {Serhii Zabolotnii and Alina Tarasova},
  title = {{An AI Assistant for Automating Anti-Corruption Expertise of Draft Legislation Based on the TRACE Framework}},
  year = {2026},
  eprint = {2611.14244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14244},
}
Figure