TL;DR
This paper introduces a novel approach to automatically correct whitespace errors in text using encoder-only Transformer models, achieving state-of-the-art quality while being over 900 times faster than previous methods. This significant speedup makes practical, large-scale whitespace correction feasible, opening doors for improved downstream NLP tasks.
What Problem Does This Paper Address?
Natural language processing (NLP) applications often rely on text being correctly segmented into words. This segmentation is typically done by splitting text at spaces. However, real-world text, especially from sources like OCR or PDF extraction, frequently contains "whitespace errors" – either spurious extra spaces or missing spaces. These errors can significantly degrade the performance of downstream NLP tasks. While previous methods have achieved good accuracy in correcting these errors, they are notoriously slow, making them impractical for processing large amounts of text. This paper aims to bridge this gap by developing a whitespace correction system that is both highly accurate and extremely fast.
Methods
The authors compare two Transformer-based architectures for whitespace correction:
- Encoder-Decoder (ED) Model: This is a more traditional approach where the model takes an input sequence and generates a corrected output sequence character by character.
- Encoder-Only (EO) Model: This is the paper's novel contribution. Instead of generating a new sequence, this model treats whitespace correction as a sequence-labeling task. For each character in the input, it predicts one of three "repair tokens": 'Keep' (K), 'Insert' (I) a space before the character, or 'Delete' (D) an existing space.
A key innovation in the EO model is its use of byte-level embeddings instead of character-level ones. This allows the model to process text without needing a predefined character vocabulary and handles unknown characters more gracefully. To manage computational efficiency, byte embeddings belonging to a single character are aggregated into a single embedding.
Both models are trained to respect existing whitespace information in the input. For inference, a sliding window approach is used to handle sequences longer than the model's fixed input size (512 tokens).
Datasets
The models are trained on a large dataset derived from Wikipedia and arXiv articles, augmented with injected OCR, spelling, and specifically, whitespace errors. The authors also use several established benchmarks for evaluation:
- Wikipedia-based: Wiki, Wiki+, Wiki+no (varying levels of errors)
- arXiv-based: arXiv OCR, arXiv pdftotext (errors from OCR and PDF extraction)
- ACL Anthology: ACL (OCR errors)
- Word Segmentation: Doval (sequences without whitespace)
- Runtime Benchmark: A custom benchmark for measuring inference speed.
Key Insights and Ideas
- Encoder-Only Dominance: The encoder-only (EO) model significantly outperforms the encoder-decoder (ED) model in terms of speed, while achieving comparable or even slightly better quality across most benchmarks. This suggests that a sequence-labeling approach is more efficient for this task.
- Byte-Level Efficiency: Using byte-level embeddings in the EO model provides a practical advantage by eliminating the need for a character vocabulary and simplifying the handling of diverse character sets, without sacrificing performance.
- Speed is Key: The most striking insight is the dramatic speedup achieved by the EO model. Being over 900 times faster than the previous best method (BID+) makes large-scale deployment practical, a hurdle that previous accurate methods couldn't overcome.
- Focal Loss for Imbalance: The EO model faces an imbalanced class distribution (many 'Keep' predictions vs. fewer 'Insert'/'Delete'). The authors found that using a focal loss function effectively addresses this, preventing the model from plateauing on the dominant class.
Key Results
- Speedup: The EO model achieves a throughput of over 200 kB/s on an A100 GPU, which is more than 900 times faster than the previous best method (BID+).
- Quality: The models achieve high sequence-averaged F1-scores, ranging from 87.5% for challenging text to 99% for text with minimal errors. The EOlarge model performs on par with or slightly better than the previous state-of-the-art BID+ model.
- Model Comparison: EOlarge outperforms EDlarge on 6 out of 7 benchmarks.
- Memory Efficiency: The models are memory-efficient, with EOlarge requiring less than 4GB of GPU memory even with a batch size of 128.
Limitations and Future Work
- Multilingualism: The current models are primarily trained and evaluated on English text. The authors acknowledge that extending this to other languages is a promising future direction, but requires the creation of new datasets and benchmarks.
- Combined Tasks: While the paper separates whitespace correction from spelling correction, recent large language models can handle both. The authors pose an open question about whether a single model can achieve both high quality and efficient inference for combined tasks.
- Model Size vs. Performance: While larger models generally improve performance, the authors observed diminishing returns with a significantly larger EO model (EOlarger), suggesting a trade-off between quality, speed, and memory. Further investigation into optimal model sizes is left for future work.
- Ambiguous Ground Truths: The authors note that in some cases, the "ground truth" for whitespace correction can be ambiguous, leading to potential discrepancies in evaluation.
Summary updated