AdminSet and AdminBERT: a Dataset and a Pre-trained Language Model to Explore the Unstructured Maze of French Administrative Documents

Thomas Sebbag, Solen Quiniou, Nicolas Stucky, Emmanuel MorinCOLING 2025

TL;DR

This paper introduces AdminBERT, the first pre-trained language model specifically designed for French administrative documents, and AdminSet, a large dataset of such documents. By fine-tuning AdminBERT on administrative data, the authors achieve state-of-the-art performance on Named Entity Recognition (NER) for this domain, demonstrating the importance of domain-specific models for handling unstructured, real-world text.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models, particularly Pre-trained Language Models (PLMs), have revolutionized text analysis. However, a significant gap exists in their application to administrative documents. These documents, while abundant and often publicly available, are characterized by their unstructured nature, formal syntax, and domain-specific terminology, making them challenging for general-purpose NLP models. This is particularly true for French administrative texts, which often lack standardized formatting. Extracting valuable information, such as named entities (organizations, people, locations), from these documents is crucial for understanding public administration and economic networks, but current models struggle to do so effectively.

Methods

The core contribution is the development of AdminBERT, a French PLM specifically pre-trained on administrative documents. This was achieved through a continuous pre-training strategy, starting with the existing French model CamemBERT-Base and further training it on the newly created AdminSet corpus. The authors also created AdminSet-NER, a labeled subset of AdminSet for Named Entity Recognition (NER) tasks.

To evaluate AdminBERT, they compared its performance against general domain language models and other specialized French NER models. This involved fine-tuning various models on the AdminSet-NER dataset and assessing their performance on NER tasks. They also explored the use of large language models (LLMs) like Mixtral 7x8B and general-purpose NER models like GLiNER.

Datasets

  • AdminSet: A newly introduced, open-source dataset comprising over 50 million text fragments (16 GB of OCR text) crawled from French public administration websites between January 2020 and December 2022. It covers a wide range of administrative contexts.
  • AdminSet-NER: A newly introduced, labeled dataset derived from AdminSet, specifically for NER tasks. It contains 814 annotated texts from official reports, public decisions, and legal decrees.
  • WikiNER: Used for evaluating the robustness of AdminBERT in a general domain.

Key Insights and Ideas

  • Domain Specificity Matters: General-purpose language models perform poorly on administrative texts, highlighting the need for domain-specific pre-training. AdminBERT significantly outperforms general models on administrative NER tasks.
  • Continuous Pre-training is Effective: Further pre-training an existing French PLM (CamemBERT) on administrative data (AdminSet) leads to substantial improvements in performance for administrative tasks.
  • Noise Tolerance is Key: The authors intentionally used OCR-processed texts with inherent noise to train models that are more robust to real-world data imperfections, which is crucial for practical applications.
  • Entity Distribution Impacts Performance: The underrepresentation of LOC entities in the AdminSet-NER dataset correlates with lower detection rates for this entity type across models.
  • Tokenizer Limitations: The tokenization of administrative names can be problematic, creating too many sub-tokens that hinder entity recognition. This suggests potential for future work in improving tokenization or incorporating metadata.

Key Results

  • AdminBERT-NER 16GB achieved the highest F1-score of 80.11% on the AdminSet-NER test set after fine-tuning, outperforming other models.
  • The lighter version, AdminBERT-NER 4GB, also performed very well with an F1-score of 79.26%.
  • General language models, even when fine-tuned, showed very low performances on administrative NER, with the best achieving only an F1-score of 12.54% (Wikineural-NER).
  • Mixtral 7x8B achieved an F1-score of 35.53%, and GLiNER achieved 53.92% on administrative NER without domain-specific fine-tuning, indicating their general capabilities but also their limitations compared to AdminBERT.
  • On a binary NER classification task, AdminBERT-NER 16GB achieved an F1-score of 92.05% for identifying named entities.
  • When fine-tuned on the general WikiNER dataset, AdminBERT 16GB achieved an F1-score of 92.00%, demonstrating its adaptability to general domains, while general models did not perform as well on administrative data.

Limitations and Future Work

  • Noise and Data Diversity: The authors acknowledge that the data might lack diversity due to the chosen time frame (2020-2022) and the focus on specific economic activities. They plan to address this in future work.
  • Tokenizer Impact: The study highlights the impact of tokenization on entity recognition and suggests investigating methods to improve tokenization or incorporate metadata to enhance model performance.
  • Further Refinement of AdminSet-NER: While AdminSet-NER is a valuable resource, the authors suggest it needs further refinement.
  • New Tasks: The authors are keen to explore applying AdminBERT to other NLP tasks within the administrative domain, such as relation extraction, to build knowledge graphs for territorial competitive intelligence.
  • Robustness to Noise: Further experiments are planned to quantify the impact of different amounts and types of noise on model training and performance.

Summary updated

BibTeX
@inproceedings{sebbag2025adminset,
  author = {Thomas Sebbag and Solen Quiniou and Nicolas Stucky and Emmanuel Morin},
  title = {{AdminSet and AdminBERT: a Dataset and a Pre-trained Language Model to Explore the Unstructured Maze of French Administrative Documents}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure