Introduction
Document forgery has become a significant global threat, with reported cases rising by nearly 16% in recent years. To mitigate this risk, verification systems employ "plausibility checks"—automated, rule-based procedures that ensure data consistency within a document. For instance, a check might verify if a document number is unique in a database or if an ID was issued on a valid working day.
Currently, these rules are manually authored by software engineers, a process that is both time-consuming and costly. While large-scale models like GPT-4 offer automation potential, security-sensitive sectors such as law enforcement are restricted by privacy regulations from uploading sensitive data to external cloud services. Moreover, legal contexts demand interpretable results rather than "black-box" predictions. This research explores adapting small, local Large Language Models (LLMs) to generate human-readable Python code for these checks within proprietary environments.
Methodology
The researchers implemented a two-phase fine-tuning strategy using Low-Rank Adaptation (LoRA) on a single Nvidia RTX 6000 GPU. This approach allows small models to learn domain-specific logic without the need for massive compute resources.
The process is divided into two distinct stages:
- Phase 1 (Code Understanding): Models were trained on a proprietary Django-based codebase (approx. 110,000 lines) and a synthetic instruction dataset. This phase established the "vocabulary" and structural logic of the target software environment.
- Phase 2 (Rule Generation): The models were further refined using expert-created plausibility checks and document property datasets. This stage focused on translating natural language requirements into functional Python code.
To address the scarcity of high-quality training data, the team utilized a larger model (Llama 3.3 70B) for data augmentation, creating synthetic examples to train the smaller Llama 3.1 8B and OpenCoder 8B models.
Datasets
The training utilized several key data sources:
- Proprietary Codebase: 110,000 lines of Python and 78 custom database models.
- Instruction Dataset: Over 21,000 synthetic task-input-output pairs.
- Plausibility Checks Dataset: 450 training chunks derived from 35 expert rules and 100 synthetic examples.
- Document Properties Dataset: 100,000 entries detailing document characteristics like issuing dates and country codes.
Key Insights
The study revealed a performance divide between specialist and generalist models. OpenCoder, pre-trained specifically on code, excelled at generating syntactically correct snippets and handling low-level logic. Conversely, Llama 3.1, a generalist model, demonstrated a superior "understanding" of real-world context, likely due to its multilingual training which aided in processing German-language database entries.
A significant challenge was the "boilerplate hurdle"—models often struggled with the initial conditional logic required to determine if a check is relevant to a specific document. To focus evaluation on the core forgery-detection logic, the researchers employed a Regex-based evaluation method to bypass minor syntax errors in these boilerplate sections.
Results and Evaluation
The impact of domain-specific fine-tuning was profound. Baseline models without adaptation were almost entirely incapable of generating executable code for the proprietary system.
Key performance highlights include:
- Phase 1 Success: After combined fine-tuning, OpenCoder Instruct achieved a 100% success rate in generating executable code for low and mid-level complexity database queries.
- Phase 2 Success: For the more complex task of generating new forgery checks from natural language, Llama 3.1 reached a 44% success rate on low-complexity tasks when using the adjusted Regex evaluation.
Limitations and Future Outlook
While promising, the current approach has limitations. The models are static; changes in the underlying codebase or the emergence of new forgery techniques require retraining. Additionally, hardware constraints limited the study to 8B-parameter models.
Future research will explore Retrieval-Augmented Generation (RAG) to allow models to dynamically reference new forgery patterns. Furthermore, integrating these programmatic checks with Computer Vision could enable real-time analysis of visual document features like fonts, seals, and layouts, creating a more holistic verification system.
Summary updated