TL;DR
This paper tackles the challenge of building a clean text corpus for Ottoman Turkish, a historically significant but NLP-understudied language. By developing novel methods for extracting and standardizing text from digitized historical periodicals, they create a valuable resource that enables further research and adaptation of modern NLP models to this historical language.
What Problem Does This Paper Address?
Natural Language Processing (NLP) has seen tremendous advancements, largely fueled by the availability of vast amounts of digitized text in modern languages. However, historical languages like Ottoman Turkish, despite their rich cultural and historical impact, suffer from a scarcity of clean, usable corpora and specialized NLP models. This lack of resources hinders scholarly research and the application of modern NLP techniques to understand this important historical language. The authors aim to bridge this gap by constructing a clean text corpus from historical Ottoman Turkish documents, which are often in complex formats like PDFs with mixed scripts and inconsistent layouts.
Methods
The authors employed a systematic approach to create their Ottoman Turkish corpus, focusing on two main steps:
- Data Extraction (Step 1): The primary data source was digitized historical periodicals in PDF format. These PDFs presented significant challenges due to:
- Mixed Scripts: Inclusion of both Latinized Turkish and Arabic letters.
- Noise: Recursive elements like dates, page numbers, and decorative elements.
-
Inconsistent Formatting: Variations in margins, layouts, and calligraphy across documents and pages.
Initially, they tried using
Pdfplumberwith rigid margin constraints, but this proved ineffective due to inconsistent formatting. They then switched toPyMuPDF, a more robust Python library. Their refined approach involved: -
Direct Text Extraction: Converting PDFs to plain text without strict margin rules.
- Regular Expression (Regex) Reformatting: Using regex to clean and structure the extracted text, ignoring illustrations and handling Arabic sentences separately. This method was chosen for its efficiency and ability to handle the unique characteristics of Ottoman Turkish without requiring complex post-OCR techniques.
- Text Standardization (Step 2): This crucial step involved normalizing the extracted text to ensure uniformity for NLP tasks. Key challenges and solutions included:
- Normalization and Mapping: Addressing occasional misinterpretations of Arabic characters as Private Use Area (PUA) characters and normalizing accented Latin letters not present in modern Turkish. They developed a mapping system to replace PUA characters with their standard Arabic equivalents and removed characters without equivalents.
- Right-to-Left (RTL) Scripts: Handling the complexity of intertwined Arabic and Latinized Turkish sentences. They implemented a method to move Arabic sentences to new lines, preserving their orientation and readability, and facilitating pattern recognition.
- Optional Translation Feature: While initially considered, direct translation of Arabic sentences was deemed too resource-intensive. The current corpus excludes Arabic texts for simplification.
Datasets
The primary dataset used in this study consists of issues from two Ottoman periodicals:
- Sebilürreşad
- Sırat-ı Müstakim
These periodicals were published between 1908 and 1925. The authors collected 25 volumes from these sources, made publicly available as images by the National Library of Turkey and as PDF documents.
For evaluation, they used a newly annotated, albeit small, Named Entity Recognition (NER) dataset for Ottoman Turkish, sourced from the Servet-i Fünun journal. This dataset contains:
- 462 training sentences
- 200 validation sentences
- 200 test sentences
Annotations were primarily for PERSON (386 instances) and LOCATION (794 instances) entities. The inter-annotator agreement (IAA) was measured at 0.82.
Key Insights and Ideas
- The "Occam's Razor" Approach to Data Extraction: The authors' decision to prioritize simplicity and computational efficiency (using Regex over complex, resource-intensive methods) for data extraction is a practical insight for working with historical documents.
- The Importance of Text Standardization: The paper highlights how crucial steps like character normalization and handling mixed scripts are for making historical texts usable for modern NLP. The PUA character issue is a specific, interesting challenge.
- Bridging the Historical-Modern NLP Gap: The core idea is that by creating a clean corpus and using it for continual pre-training of existing models (like BERTurk), we can adapt powerful modern NLP tools to historical languages, unlocking new research avenues.
- The Trade-off in Handling Arabic Text: The decision to exclude Arabic text for simplification, despite its potential contextual value, is a pragmatic choice driven by resource constraints and the focus on Turkish text processing.
Key Results
- Corpus Creation: The paper successfully constructed a 17M-token corpus of clean Ottoman Turkish texts from historical periodicals.
- NER Performance Improvement: Continual pre-training of the BERTurk model on the newly created Ottoman Turkish corpus led to a slight improvement in NER performance on a custom Ottoman Turkish NER dataset.
- BERTurk + PT (further pre-trained) achieved an F1 score of 0.858, compared to the base BERTurk model's 0.850.
- Precision and Recall also saw marginal gains with further pre-training.
- Ablation Study Insights: An ablation study showed that the NER model reached peak performance with approximately 50% of the pre-training data, suggesting a saturation point or potential overfitting with larger amounts of data.
Limitations and Future Work
- Limitations:
- Timeframe Bias: The corpus relies on periodicals from a specific time frame (1908-1925), which might introduce biases in language usage and topics.
- Inherent Errors: Despite efforts, there may be residual errors or omissions in the preprocessing and extraction steps.
- Corpus Size: The current corpus size (17M tokens) is considered too small for pre-training a transformer model from scratch, potentially limiting scalability and generalizability.
- Arabic Text Exclusion: The exclusion of Arabic text, while simplifying the process, might lead to gaps in contextual meaning.
- Future Work:
- Corpus Expansion: Expanding the corpus with more diverse sources and timeframes.
- Refining Preprocessing: Improving techniques for handling Arabic text and other complex linguistic features.
- Exploring Additional NLP Tasks: Applying the corpus and models to other NLP tasks beyond NER.
- Investigating Arabic Text Integration: Developing more efficient methods to integrate or translate Arabic sentences to enrich the corpus's contextual information.
- Scalability and Generalizability: Further research into how the model's performance scales with larger datasets and its generalizability across different Ottoman Turkish texts.
Summary updated