Restoring Missing Spaces in Scraped Hebrew Social Media

Avi Shmidman, Shaltiel ShmidmanNAACL 2025

TL;DR

Scraped Hebrew social media text often has missing spaces, making it hard to process. This paper introduces a new character-based BERT model for Hebrew that effectively restores these spaces, achieving high accuracy and speed, making it practical for large-scale analysis.

What Problem Does This Paper Address?

Social media data, when scraped from the web, frequently suffers from missing spaces between words. This happens because web scraping algorithms use heuristics to decide where to insert spaces, and these heuristics aren't perfect. For languages like Hebrew, which are written without vowels, this problem is particularly acute. The lack of vowels means a sequence of letters can be interpreted in multiple ways, making it difficult to simply use a dictionary to split conflated words. For example, "קושיכל" could be "קושי כל" (difficulty everyone) or "קו שיכל" (line transposed). These missing spaces can significantly impair downstream NLP tasks like parsing, segmentation, and information retrieval, and also propagate errors into language models trained on such data.

Methods

The authors propose and evaluate several neural approaches. They start by examining existing Hebrew encoder models (like mBERT, AlephBERT) that use wordpiece tokenizers, finding them ill-suited for character-level tasks due to their loss of fine-grained information. They then consider TavBERT, which uses a character-based representation but was trained with a SpanBERT objective, making it less effective for single-character predictions.

The core contribution is a new character-based BERT model for Hebrew, dubbed DictaBERT-char. This model is pretrained using a masked language modeling objective at the character level, specifically designed to predict single masked characters. This approach is hypothesized to be more robust for character-level tasks. The pretraining uses a purely character-level tokenizer and a consistent context length of 2048 tokens.

They also explore decoder-based models (LLMs), specifically fine-tuning an open-weight LLM (Dicta-LM 2.0) and evaluating prompt engineering with proprietary LLMs (GPT-40 and GPT-40-Mini).

Datasets

  • Training Data: A collection of 150,000 Hebrew sentences from high-quality corpora, with 1-4 random spaces removed from 20% of them.
  • Test Corpus: 6,000 sentences from the newly-released FineWeb2 corpus, carefully selected to avoid overlap with previous corpora. Spaces were removed only between words (not near punctuation) to focus on the challenging conflation problem.
  • OSCAR Internet Crawl: Used as the source for input examples in the qualitative analysis section.

Key Insights and Ideas

  • Character-level BERT is crucial for Hebrew space restoration: Standard wordpiece tokenizers obscure essential character-level information, making them suboptimal for this task.
  • Single-character prediction is key: Unlike SpanBERT objectives, training a model to predict individual characters is more effective for fine-grained tasks like space restoration.
  • Class imbalance is a significant challenge: In the training data, correctly spaced sentences far outnumber those with missing spaces, requiring careful downsampling strategies during training.
  • LLMs can perform the task, but with trade-offs: While LLMs show promise, their computational cost and inference speed make them less practical for large-scale social media corpora compared to a fine-tuned encoder model.
  • Post-processing filters can mitigate failures: The model's tendency to over-split words or misinterpret grammatical suffixes can be addressed with additional filtering steps, such as using NER models or rule-based checks for suffixes.

Key Results

  • DictaBERT-char (Our Model) achieves state-of-the-art performance: At a confidence threshold of 0.9, it achieves an F1 score of 0.968, rivaling the best decoder-based model (DictaLM 2.0 at 0.976) while being significantly faster.
  • Speed advantage: DictaBERT-char runs approximately 30 times faster than DictaLM 2.0 with guidance, making it highly practical for real-world corpora.
  • DictaLM 2.0 (fine-tuned) is the best decoder model: It achieves a high F1 score of 0.976.
  • GPT-40 performs well but is slower: It achieves a high F1 score (0.961) but is much slower than the custom BERT model. GPT-40-Mini performs less well.
  • Quantized DictaLM 2.0 (AWQ) is efficient: It requires only 5GB of VRAM and performs nearly as well as the full-precision model.

Limitations and Future Work

  • Failures due to typos and proper nouns: The model sometimes struggles when input text contains additional typos beyond missing spaces, or when it misinterprets proper nouns, attempting to split them.
  • Failures due to unusual grammatical suffixes: The model can be fooled by long words with rare grammatical suffixes, leading to incorrect space insertions.
  • Future Work: The authors suggest using additional filters, such as NER models to identify proper names and prevent splitting, or scripts to check for Hebrew grammatical suffixes, to further refine the model's output and restrain it from over-splitting. They also release their pretrained character-based BERT model and the fine-tuned space-restoration model to the community.

Summary updated

BibTeX
@inproceedings{shmidman2025restoring,
  author = {Avi Shmidman and Shaltiel Shmidman},
  title = {{Restoring Missing Spaces in Scraped Hebrew Social Media}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure