TL;DR
The authors introduce AIMSCheck, a multi-level, human-in-the-loop AI framework that leverages fine-tuned and prompted Large Language Models (LLMs) to automate the compliance review of corporate modern slavery statements. Alongside the framework, the paper introduces two newly annotated datasets from the UK and Canada, demonstrating that NLP models trained on Australian compliance data generalize remarkably well across international legal jurisdictions.
What Problem Does This Paper Address?
Modern Slavery Acts (MSAs) in countries like the UK, Australia, and Canada legally mandate large corporations to publish annual disclosures detailing their efforts to eliminate forced labor from their supply chains. However, monitoring compliance is an operational nightmare: tens of thousands of multi-page, unstructured documents are published yearly, filled with corporate jargon and vague assertions.
Prior natural language processing (NLP) attempts have been severely bottlenecked by a scarcity of annotated data. Furthermore, because compliance criteria vary slightly by country, it was previously unknown whether an AI model trained on one country's legal framework could effectively audit disclosures in another.
Methods
The authors developed AIMSCheck, an end-to-end framework designed to assist human analysts by breaking compliance validation down into three distinct levels:
- Sentence-Level Prediction: Classifies text based on its relevance to nine common compliance criteria. The authors benchmarked open models (BERT, Llama 3.2 3B) fine-tuned on Australian data against massive commercial models (GPT-4o, DeepSeek-R1) using zero-shot, few-shot, and Chain-of-Thought (CoT) prompting. Input sentences were evaluated with a balanced 100-word context window to capture surrounding meaning.
- Token-Level Explanation: Integrates SHAP (SHapley Additive exPlanations) to calculate the exact impact of individual words on the model's final prediction, providing crucial transparency for human auditors.
- Evidence Status Tracking: Goes beyond simple relevance by categorizing the status of corporate claims. It uses an NLTK-based tense classifier to identify future commitments (e.g., "we plan to train...") and a zero-shot BART-MNLI model to flag negative evidence or explicit admissions of inaction.
Datasets
The study utilizes three distinct datasets to enable cross-jurisdictional evaluation:
- AIMS.au: An existing dataset containing sentence-level multi-label annotations aligned with the Australian MSA, used here as the primary training set.
- AIMS.uk (New): 50 multi-page corporate statements sourced from the UK government registry, manually annotated by a domain expert (2,807 total sentences).
- AIMS.ca (New): 50 multi-page corporate statements sourced from the Canadian government registry, manually annotated by a domain expert (3,658 total sentences).
To make cross-border evaluation possible, domain experts mapped the distinct legal requirements of each country into nine unified "Common Criteria" (such as Approval, Risk Identification, and Mitigation).
Key Insights and Ideas
- Strong Cross-Border Generalization: A key "aha" moment was discovering that models trained only on Australian corporate disclosures perform robustly when evaluated on UK and Canadian documents.
- Minimal Vocabulary Shift: To understand why the models generalized so well, the authors analyzed the Jensen-Shannon (JS) divergence between the datasets. They found that despite minor domain shifts due to unique regional company and supplier names, the core vocabulary used to discuss modern slavery compliance remains highly stable globally.
- The Granularity Trap: Both fine-tuned models and large LLMs struggle to differentiate between closely related, overlapping criteria—specifically when trying to separate descriptions of a business's "structure" from its "operations."
Key Results
- Fine-Tuning Beats Zero-Shot Giants: Smaller, fine-tuned models consistently outperformed massive zero-shot and few-shot LLMs.
- Top Performer: A fine-tuned Llama 3.2 3B model utilizing 100 words of context achieved the highest overall F1 scores across all test sets: 0.738 on AIMS.au, 0.719 on AIMS.ca, and 0.686 on AIMS.uk.
- Prompt Engineering Progression: For GPT-4o, adding Chain-of-Thought (CoT) reasoning and few-shot examples steadily improved performance, moving from a baseline zero-shot F1 of 0.601 to a few-shot CoT F1 of 0.617 on the Australian dataset.
- Evidence Tracking Success: The framework achieved macro-F1 scores of 66.83% (UK) and 57.83% (Canada) for identifying future commitments, proving it can successfully track corporate promises over time.
Limitations and Future Work
While AIMSCheck marks a major step forward, the authors note several limitations:
- Dataset Scale: The newly introduced UK and Canadian evaluation sets consist of only 50 statements each, representing a small fraction of total annual disclosures.
- Layout and Preprocessing Vulnerabilities: The text parsing and OCR pipeline occasionally struggled with complex document layouts, leading to broken sentences and poorly extracted bulleted lists that confused the classifiers.
- Language Constraints: All datasets and models evaluated in this study are exclusively in English, leaving multilingual expansion (a necessity for global supply chains) as an open avenue for future research.
Summary updated