Abstract
Automating supply chain monitoring requires the translation of natural language legal contracts into formal specification languages. This paper explores the efficacy of Large Language Models (LLMs) in performing this extraction. We propose a two-step prompting pipeline to generate clauses in the $\mathcal{L}_c$ language. While models like ChatGPT-3.5 and Llama-3 exhibit high generative performance, our findings reveal a critical "verification gap" in neuro-symbolic frameworks, where models struggle to reliably validate the logical consistency of their own outputs.
Introduction
In modern supply chain management, a contract is more than a legal document; it is a set of executable obligations specifying responsibilities, goals, and temporal constraints. To enable automated monitoring by autonomous agents, these obligations must be represented in formal logic. However, the manual encoding of complex legal prose into formal languages like $\mathcal{L}_c$ is a grueling, error-prone task that hinders the scalability of automated systems.
As illustrated in the system architecture above, the "Contract Specification" serves as a foundational input for reasoning, planning, and diagnosis within a monitoring framework. This research investigates whether LLMs can bridge the "Knowledge Acquisition" gap by automatically extracting these specifications from natural language text.
Methodology
The study employs a modular two-step prompting strategy to translate synthetic supply chain contracts into formal $\mathcal{L}_c$ clauses.
Two-Step Prompting Pipeline
- Information Extraction: The LLM is prompted to identify key entities, including suppliers, buyers, products, quantities, and temporal constraints.
- Clause Construction: Using the extracted metadata and original text, the model generates formal clauses following the syntax:
ref_id : agent responsible_for goal when time_expression.
Experimental Setup
The researchers evaluated ChatGPT-3.5-turbo and a locally hosted Llama-3 (via Ollama). To mitigate hallucinations and improve reliability, the DSPy framework was utilized. Specifically, a ChainOfThought module was implemented to create a verification layer intended to justify whether a generated clause is consistent with the source text.
Dataset
Due to the lack of standard benchmarks for this specific task, a custom dataset of 11 synthetic supply chain contracts was developed. These range from home building to pharmaceutical manufacturing, each accompanied by a "ground truth" set of formal clauses for evaluation.
Results and Discussion
The experiments yielded a dichotomy between generative capability and verification reliability.
Generative Performance
Both ChatGPT-3.5 and Llama-3 demonstrated high accuracy in extracting contract details and formatting them into $\mathcal{L}_c$. Most generated clauses closely matched the ground truth, suggesting that LLMs are highly capable of the primary translation task.
The Verification Gap
A significant finding was the "No Free Lunch" of verification. While LLMs excel at generation, they proved dangerously over-confident during self-verification. The DSPy-based verification module frequently validated incorrect or intentionally corrupted clauses with a "Yes" response.
However, verification performance improved dramatically when the task granularity was reduced. The system was highly accurate at identifying simple entity mismatches (e.g., misspelled names or incorrect quantities) but struggled with the holistic logic of complex multi-part clauses.
Conclusion
LLMs represent a promising path toward automating the acquisition of formal contract specifications. While they can effectively "write the rules," current neuro-symbolic tools require more advanced logic-handling capabilities to reliably "check the math." Future work will focus on expanding dataset diversity and refining validation modules to handle the nuances of formal logical verification more robustly.
Summary updated