Generation of Programmatic Rules for Document Forgery Detection Using Large Language Models

Valentin Schmidberger, Manuel Eberhardinger, Setareh Maghsudi, Johannes MaucherarXiv 2025 · DOI · arXiv

Introduction

Document forgery has become a significant global threat, with reported cases rising by nearly 16% in recent years. To mitigate this risk, verification systems employ "plausibility checks"—automated, rule-based procedures that ensure data consistency within a document. For instance, a check might verify if a document number is unique in a database or if an ID was issued on a valid working day.

Currently, these rules are manually authored by software engineers, a process that is both time-consuming and costly. While large-scale models like GPT-4 offer automation potential, security-sensitive sectors such as law enforcement are restricted by privacy regulations from uploading sensitive data to external cloud services. Moreover, legal contexts demand interpretable results rather than "black-box" predictions. This research explores adapting small, local Large Language Models (LLMs) to generate human-readable Python code for these checks within proprietary environments.

Methodology

The researchers implemented a two-phase fine-tuning strategy using Low-Rank Adaptation (LoRA) on a single Nvidia RTX 6000 GPU. This approach allows small models to learn domain-specific logic without the need for massive compute resources.

Overview of the Document Verification System and the two-phase fine-tuning workflow.

The process is divided into two distinct stages:

  1. Phase 1 (Code Understanding): Models were trained on a proprietary Django-based codebase (approx. 110,000 lines) and a synthetic instruction dataset. This phase established the "vocabulary" and structural logic of the target software environment.
  2. Phase 2 (Rule Generation): The models were further refined using expert-created plausibility checks and document property datasets. This stage focused on translating natural language requirements into functional Python code.

To address the scarcity of high-quality training data, the team utilized a larger model (Llama 3.3 70B) for data augmentation, creating synthetic examples to train the smaller Llama 3.1 8B and OpenCoder 8B models.

Datasets

The training utilized several key data sources:

  • Proprietary Codebase: 110,000 lines of Python and 78 custom database models.
  • Instruction Dataset: Over 21,000 synthetic task-input-output pairs.
  • Plausibility Checks Dataset: 450 training chunks derived from 35 expert rules and 100 synthetic examples.
  • Document Properties Dataset: 100,000 entries detailing document characteristics like issuing dates and country codes.

Key Insights

The study revealed a performance divide between specialist and generalist models. OpenCoder, pre-trained specifically on code, excelled at generating syntactically correct snippets and handling low-level logic. Conversely, Llama 3.1, a generalist model, demonstrated a superior "understanding" of real-world context, likely due to its multilingual training which aided in processing German-language database entries.

A significant challenge was the "boilerplate hurdle"—models often struggled with the initial conditional logic required to determine if a check is relevant to a specific document. To focus evaluation on the core forgery-detection logic, the researchers employed a Regex-based evaluation method to bypass minor syntax errors in these boilerplate sections.

Results and Evaluation

The impact of domain-specific fine-tuning was profound. Baseline models without adaptation were almost entirely incapable of generating executable code for the proprietary system.

Performance comparison of models across different complexity levels and training datasets.

Key performance highlights include:

  • Phase 1 Success: After combined fine-tuning, OpenCoder Instruct achieved a 100% success rate in generating executable code for low and mid-level complexity database queries.
  • Phase 2 Success: For the more complex task of generating new forgery checks from natural language, Llama 3.1 reached a 44% success rate on low-complexity tasks when using the adjusted Regex evaluation.

Limitations and Future Outlook

While promising, the current approach has limitations. The models are static; changes in the underlying codebase or the emergence of new forgery techniques require retraining. Additionally, hardware constraints limited the study to 8B-parameter models.

Future research will explore Retrieval-Augmented Generation (RAG) to allow models to dynamically reference new forgery patterns. Furthermore, integrating these programmatic checks with Computer Vision could enable real-time analysis of visual document features like fonts, seals, and layouts, creating a more holistic verification system.

Summary updated

BibTeX
@misc{schmidberger2025generation,
  author = {Valentin Schmidberger and Manuel Eberhardinger and Setareh Maghsudi and Johannes Maucher},
  title = {{Generation of Programmatic Rules for Document Forgery Detection Using Large Language Models}},
  year = {2025},
  eprint = {2512.19228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.19228},
}
Figure