ArcGPT: A Large Language Model Tailored for Real-world Archival Applications

Shitou Zhang, Jingrui Hou, Siyuan Peng, Zuchao Li, Qibiao Hu, Ping WangarXiv 2023 · DOI · arXiv

Introduction

Archives serve as the fundamental backbone of institutional memory and historical preservation. However, the rapid digitization of records has created a significant data management challenge; for instance, electronic archives in China reached over 1,600 Terabytes by 2021. Manually processing this volume of data for retention, confidentiality, and public access is increasingly untenable. While general-purpose Large Language Models (LLMs) like GPT-4 have shown promise in broad natural language tasks, they often struggle with the specialized administrative jargon, historical linguistic variations, and complex regulatory frameworks inherent in archival science. To address this, we introduce ArcGPT, the first LLM specifically pre-trained for the archival domain, alongside AMBLE, a comprehensive benchmark for evaluating AI in real-world archival workflows.

Methodology

ArcGPT is a 7-billion parameter model built upon the BatGPT architecture. Its development focused on three core pillars:

  1. Domain-Specific Pre-training: The model was exposed to a massive corpus of archival data, including professional journals and historical records. This allows ArcGPT to internalize the evolution of administrative language over decades.
  2. Contextual Awareness: By training on actual archival records, the model learns the specific phraseology and structural patterns used by professional archivists.
  3. Instruction Tuning: ArcGPT underwent fine-tuning to follow complex instructions related to archival tasks, such as determining document sensitivity or predicting preservation timelines.

The AMBLE Benchmark

To rigorously evaluate performance, we developed the Archival Multi-task Benchmark for LLM Evaluation (AMBLE). This dataset comprises 14,469 training samples and 1,000 test samples, all meticulously labeled by archival science experts. AMBLE targets four critical tasks: Retention Period Prediction, Open Access Identification, Confidentiality Prediction, and Post-OCR Processing.

Table 1: Statistics of the AMBLE dataset across four primary archival tasks.

Performance Evaluation

Experimental results demonstrate that ArcGPT significantly outperforms other generative models in archival contexts. In classification-heavy tasks like confidentiality prediction, ArcGPT achieved an F1 score of 94.40, surpassing general-purpose models like ChatGLM and Chinese-LLaMA-Alpaca.

Table 2: Comparative performance of ArcGPT against other models on retention, open access, and confidentiality tasks.

While ArcGPT is the leading generative model, the evaluation revealed a "performance gap" where smaller, task-specific predictive models (such as RoBERTa) still maintain a slight edge in pure classification accuracy. Furthermore, in Post-OCR processing—the task of correcting text from noisy scans—ArcGPT proved competitive with other LLMs but trailed behind specialized Chinese Spelling Correction (CSC) models like Mengzi-T5. This suggests that while LLMs excel at contextual understanding, specialized tools remain superior for low-level text cleaning.

Conclusion and Future Work

ArcGPT represents a significant advancement in applying generative AI to specialized administrative domains. By bridging the gap between general linguistic capability and archival expertise, it provides a scalable solution for modern record management. Future research will focus on optimizing the model's OCR correction capabilities and further refining its generative architecture to match the precision of specialized predictive classifiers. The authors encourage the archival community to integrate ArcGPT into active workflows to facilitate iterative, real-world improvement.

Summary updated

Abstract

Archives play a crucial role in preserving information and knowledge, and the exponential growth of such data necessitates efficient and automated tools for managing and utilizing archive information resources. Archival applications involve managing massive data that are challenging to process and analyze. Although LLMs have made remarkable progress in diverse domains, there are no publicly available archives tailored LLM. Addressing this gap, we introduce ArcGPT, to our knowledge, the first general-purpose LLM tailored to the archival field. To enhance model performance on real-world archival tasks, ArcGPT has been pre-trained on massive and extensive archival domain data. Alongside ArcGPT, we release AMBLE, a benchmark comprising four real-world archival tasks. Evaluation on AMBLE shows that ArcGPT outperforms existing state-of-the-art models, marking a substantial step forward in effective archival data management. Ultimately, ArcGPT aims to better serve the archival community, aiding archivists in their crucial role of preserving and harnessing our collective information and knowledge.
BibTeX
@misc{zhang2023arcgpt,
  author = {Shitou Zhang and Jingrui Hou and Siyuan Peng and Zuchao Li and Qibiao Hu and Ping Wang},
  title = {{ArcGPT: A Large Language Model Tailored for Real-world Archival Applications}},
  year = {2023},
  eprint = {2307.14852},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.14852},
}
Figure