TL;DR
This paper introduces a novel, unsupervised method for automatically correcting common whitespace errors in digitized historical texts. By leveraging Google n-gram statistics, their approach effectively identifies and fixes concatenated words, significantly improving the accuracy of historical document analysis without requiring manual annotation or pre-defined vocabularies.
What Problem Does This Paper Address?
Digitizing historical texts, especially from sources like 19th-century newspapers, is crucial for humanities research. However, the process, whether through optical character recognition (OCR) or manual transcription, often introduces errors. One particularly pervasive issue is "whitespace errors," where words are incorrectly joined together (e.g., "senatoradmits" instead of "senator admits"). These errors, also known as word segmentation errors, can severely hinder downstream analyses like topic modeling and word embeddings. While manual correction is an option, it's prohibitively expensive for large corpora. Existing methods often rely on predefined vocabularies, which are insufficient for historical texts with evolving language and unique terms. This paper aims to bridge this gap by providing an unsupervised, scalable solution to fix these segmentation errors.
Methods
The core of the proposed method lies in using large-scale n-gram counts, specifically from Google Books, to infer the likelihood of a word being segmented. The technique is based on a likelihood ratio test.
- Unsupervised Token Segmentation: The system identifies potential whitespace errors by looking for tokens that are "out-of-vocabulary" but can be broken down into smaller, "in-vocabulary" substrings.
- Likelihood Ratio: The probability of a proposed segmentation is compared against the probability of the original, unsegmented token. This is done using n-gram probabilities (unigrams and bigrams).
- Contextual Likelihood Ratio: To improve accuracy, the method incorporates contextual information by considering the surrounding words (trigrams and bigrams). This helps disambiguate cases where a segmentation might seem plausible in isolation but is unlikely in context (e.g., "often" vs. "of ten").
- Interpolation: To handle sparse counts in n-grams, especially for less frequent words or contexts, the paper employs interpolated probabilities. This blends empirical n-gram counts with lower-order n-gram probabilities (or unigrams) to provide more robust estimates.
Datasets
- Google n-grams: Used for obtaining large-scale n-gram counts for training and probability estimation. The paper notes that it's not essential for this corpus to be perfectly clean, as long as errors are independent and identically distributed.
- Accessible Archives corpus: A manually digitized corpus of 19th-century US newspapers. This corpus was used for evaluating the proposed method.
- Annotated subset: A subset of 200 terms from the Accessible Archives corpus, manually annotated by two of the authors, was used to create a benchmark for evaluating segmentation accuracy. This subset focused on terms appearing at least 5 times in the corpus.
Key Insights and Ideas
- N-gram statistics are powerful for historical text: The paper demonstrates that even without explicit dictionaries of historical terms, large-scale n-gram data can effectively capture linguistic patterns and help resolve segmentation ambiguities.
- Context is crucial for disambiguation: Simply looking at individual word probabilities isn't enough. Considering the surrounding words significantly improves the ability to correctly segment or not segment a token.
- Interpolation smooths out sparse data: For historical texts, many word combinations will have low counts. Interpolation is essential for making reliable probability estimates and preventing errors due to insufficient data.
- Whitespace errors are a distinct problem: The authors argue that whitespace errors, while a form of "noise," can be addressed by more lightweight, unsupervised methods than general OCR post-correction techniques, especially when dealing with manually keyed-in data that still suffers from such errors.
Key Results
The proposed contextualized and interpolated method achieved impressive results:
- High Recall: The system achieved 77% recall at a false positive rate of less than 1%, and 91% recall at a false positive rate of less than 3%.
- Improved Accuracy: Contextualization, especially when combined with interpolation, significantly improved recall at low false positive rates compared to non-contextual methods.
- Qualitative Success: The method correctly segmented examples like "Themotion" to "The motion," "andprovided" to "and provided," and "wearthese" to "wear these." It also successfully avoided erroneous segmentations for tokens like "innumerous" (correctly kept as is) and "Safeguard" (correctly kept as is).
Limitations and Future Work
- Limited to two segments: The current method is designed to handle cases where a token needs to be split into at most two segments. More complex segmentation scenarios are left for future work.
- Language dependency: The current approach is demonstrated on English texts. Future work should explore its applicability to other languages.
- Other error types: The paper focuses specifically on whitespace errors. Future work could investigate extending these techniques to address other types of digitization errors.
- Tuning interpolation coefficients: The authors suggest that further performance improvements might be achieved by more systematically tuning the interpolation coefficients on a validation set.
Summary updated