AIMSCheck: Leveraging LLMs for AI-Assisted Review of Modern Slavery Statements Across Jurisdictions

Adriana Eufrosina Bora, Akshatha Arodi, Duoyi Zhang, Jordan Bannister, Mirko Bronzi, Arsene Fansi Tchango, Md Abul Bashar, Richi Nayak, Kerrie Mengersen

TL;DR

The authors introduce AIMSCheck, a multi-level, human-in-the-loop AI framework that leverages fine-tuned and prompted Large Language Models (LLMs) to automate the compliance review of corporate modern slavery statements. Alongside the framework, the paper introduces two newly annotated datasets from the UK and Canada, demonstrating that NLP models trained on Australian compliance data generalize remarkably well across international legal jurisdictions.

What Problem Does This Paper Address?

Modern Slavery Acts (MSAs) in countries like the UK, Australia, and Canada legally mandate large corporations to publish annual disclosures detailing their efforts to eliminate forced labor from their supply chains. However, monitoring compliance is an operational nightmare: tens of thousands of multi-page, unstructured documents are published yearly, filled with corporate jargon and vague assertions.

Prior natural language processing (NLP) attempts have been severely bottlenecked by a scarcity of annotated data. Furthermore, because compliance criteria vary slightly by country, it was previously unknown whether an AI model trained on one country's legal framework could effectively audit disclosures in another.

Methods

The authors developed AIMSCheck, an end-to-end framework designed to assist human analysts by breaking compliance validation down into three distinct levels:

  1. Sentence-Level Prediction: Classifies text based on its relevance to nine common compliance criteria. The authors benchmarked open models (BERT, Llama 3.2 3B) fine-tuned on Australian data against massive commercial models (GPT-4o, DeepSeek-R1) using zero-shot, few-shot, and Chain-of-Thought (CoT) prompting. Input sentences were evaluated with a balanced 100-word context window to capture surrounding meaning.
  2. Token-Level Explanation: Integrates SHAP (SHapley Additive exPlanations) to calculate the exact impact of individual words on the model's final prediction, providing crucial transparency for human auditors.
  3. Evidence Status Tracking: Goes beyond simple relevance by categorizing the status of corporate claims. It uses an NLTK-based tense classifier to identify future commitments (e.g., "we plan to train...") and a zero-shot BART-MNLI model to flag negative evidence or explicit admissions of inaction.
Figure 2: The AIMSCheck pipeline is designed to process modern slavery statements mandated by the Modern Slavery Acts of Australia, the United Kingdom and Canada. It generates predictions at sentence-level across multiple reporting criteria, complemented by token-level explainability techniques and evidence tracking. These outputs enable human analysts to efficiently and thoroughly review company compliance.

Datasets

The study utilizes three distinct datasets to enable cross-jurisdictional evaluation:

  • AIMS.au: An existing dataset containing sentence-level multi-label annotations aligned with the Australian MSA, used here as the primary training set.
  • AIMS.uk (New): 50 multi-page corporate statements sourced from the UK government registry, manually annotated by a domain expert (2,807 total sentences).
  • AIMS.ca (New): 50 multi-page corporate statements sourced from the Canadian government registry, manually annotated by a domain expert (3,658 total sentences).

To make cross-border evaluation possible, domain experts mapped the distinct legal requirements of each country into nine unified "Common Criteria" (such as Approval, Risk Identification, and Mitigation).

Figure 1: The common and unique mandatory reporting criteria associated with the Modern Slavery Acts in Australia, the United Kingdom, and Canada. The three datasets (AIMS.au, AIMS.uk, AIMS.ca) allow for cross- jurisdictional evaluation of compliance checking tools.

Key Insights and Ideas

  • Strong Cross-Border Generalization: A key "aha" moment was discovering that models trained only on Australian corporate disclosures perform robustly when evaluated on UK and Canadian documents.
  • Minimal Vocabulary Shift: To understand why the models generalized so well, the authors analyzed the Jensen-Shannon (JS) divergence between the datasets. They found that despite minor domain shifts due to unique regional company and supplier names, the core vocabulary used to discuss modern slavery compliance remains highly stable globally.
  • The Granularity Trap: Both fine-tuned models and large LLMs struggle to differentiate between closely related, overlapping criteria—specifically when trying to separate descriptions of a business's "structure" from its "operations."

Key Results

  • Fine-Tuning Beats Zero-Shot Giants: Smaller, fine-tuned models consistently outperformed massive zero-shot and few-shot LLMs.
  • Top Performer: A fine-tuned Llama 3.2 3B model utilizing 100 words of context achieved the highest overall F1 scores across all test sets: 0.738 on AIMS.au, 0.719 on AIMS.ca, and 0.686 on AIMS.uk.
  • Prompt Engineering Progression: For GPT-4o, adding Chain-of-Thought (CoT) reasoning and few-shot examples steadily improved performance, moving from a baseline zero-shot F1 of 0.601 to a few-shot CoT F1 of 0.617 on the Australian dataset.
  • Evidence Tracking Success: The framework achieved macro-F1 scores of 66.83% (UK) and 57.83% (Canada) for identifying future commitments, proving it can successfully track corporate promises over time.

Limitations and Future Work

While AIMSCheck marks a major step forward, the authors note several limitations:

  • Dataset Scale: The newly introduced UK and Canadian evaluation sets consist of only 50 statements each, representing a small fraction of total annual disclosures.
  • Layout and Preprocessing Vulnerabilities: The text parsing and OCR pipeline occasionally struggled with complex document layouts, leading to broken sentences and poorly extracted bulleted lists that confused the classifiers.
  • Language Constraints: All datasets and models evaluated in this study are exclusively in English, leaving multilingual expansion (a necessity for global supply chains) as an open avenue for future research.

Summary updated

Abstract

Modern Slavery Acts mandate that corporations disclose their efforts to combat modern slavery, aiming to enhance transparency and strengthen practices for its eradication. However, verifying these statements remains challenging due to their complex, diversified language and the sheer number of statements that must be reviewed. The development of NLP tools to assist in this task is also difficult due to a scarcity of annotated data. Furthermore, as modern slavery transparency legislation has been introduced in several countries, the generalizability of such tools across legal jurisdictions must be studied. To address these challenges, we work with domain experts to make two key contributions. First, we present AIMS.uk and AIMS.ca, newly annotated datasets from the UK and Canada to enable cross-jurisdictional evaluation. Second, we introduce AIMSCheck, an end-to-end framework for compliance validation. AIMSCheck decomposes the compliance assessment task into three levels, enhancing interpretability and practical applicability. Our experiments show that models trained on an Australian dataset generalize well across UK and Canadian jurisdictions, demonstrating the potential for broader application in compliance monitoring. We release the benchmark datasets and AIMSCheck to the public to advance AI-adoption in compliance assessment and drive further research in this field.
BibTeX
@inproceedings{bora2025aimscheck,
  author = {Adriana Eufrosina Bora and Akshatha Arodi and Duoyi Zhang and Jordan Bannister and Mirko Bronzi and Arsene Fansi Tchango and Md Abul Bashar and Richi Nayak and Kerrie Mengersen},
  title = {{AIMSCheck: Leveraging LLMs for AI-Assisted Review of Modern Slavery Statements Across Jurisdictions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure