MsBERT: A New Model for the Reconstruction of Lacunae in Hebrew Manuscripts

Avi Shmidman, Ometz Shmidman, Hillel Gershuni, Moshe KoppelACL 2024

TL;DR

Researchers have developed MsBERT, a specialized BERT model designed to reconstruct missing text in ancient Hebrew manuscripts. Unlike existing models trained on modern Hebrew, MsBERT is pre-trained on a unique corpus of historical Hebrew texts and manuscript transcriptions, significantly outperforming previous approaches in predicting missing words and identifying textual citations.

What Problem Does This Paper Address?

Ancient Hebrew manuscripts are invaluable historical documents, but they are often damaged, making their texts incomplete and difficult to decipher. Scholars spend considerable effort reconstructing these lacunae (gaps). While general-purpose BERT models for Hebrew exist, they struggle with the unique orthographic variations found in manuscripts, such as inconsistent use of vowel letters (matres lectionis), abbreviations, and truncated words. These deviations, which are rampant rather than occasional, mean that even models trained on historical printed texts (which follow more standardized orthography) are not well-equipped for the task. This paper addresses the need for a model specifically tailored to the nuances of Hebrew manuscript text.

Methods

The core of this work is the development of MsBERT, a new BERT model. Key methodological aspects include:

  • Custom Tokenizer: A new word-piece tokenizer was trained specifically for Hebrew manuscript texts. This tokenizer was adjusted to handle apostrophes and double-quote marks more effectively, preventing them from being split into separate tokens. Crucially, two special tokens, [GAP] (for large gaps) and [ONEGAP] (for single missing words), were added to the vocabulary to explicitly represent textual lacunae.
  • Multi-Stage Training: MsBERT was trained in three phases to balance the need for manuscript-specific data with the benefits of larger corpora:
  • Phase 1: Focused on manuscript transcriptions and pre-print Rabbinic texts to build initial manuscript-specific understanding.
  • Phase 2: Incorporated a larger, comprehensive Rabbinic corpus alongside the Phase 1 data to broaden vocabulary and linguistic coverage.
  • Phase 3: A final, crucial phase where the model was exclusively trained on the Hebrew manuscript corpus to fine-tune its performance on the specific orthographic challenges.
  • Masked Language Modeling (MLM): The standard BERT MLM objective was used, but with specific constraints: masking was avoided for partial word-pieces (to ensure meaningful prediction tasks) and for the special [GAP] and [ONEGAP] tokens (to train the model to predict actual words).

Datasets

The training and evaluation of MsBERT utilized several corpora:

  • Hebrew Manuscript Corpus: A newly compiled corpus of over 67 million words from transcriptions provided by Hebrew Studies scholars, covering texts from the 3rd to 13th centuries.
  • Pre-Print Rabbinic Corpus: A collection of digitized Rabbinic texts predating the 15th century (49 million words).
  • Comprehensive Rabbinic Corpus: A large corpus of over 400 million words, including texts from Sefaria and in-house digitized sources, spanning from the 3rd century to the present.

For evaluation, the authors used manuscript transcriptions from two genres not included in the training data:

  • A homiletic text from the 5th-6th century (Shir Hashirim Rabba).
  • A legal text from the 4th quarter of the first millennium (Me'en Shiltot).

Additionally, for citation identification, they used transcriptions of Mekhilta de-Rashbi and Shir Hashirim Rabah, contrasted with training data from printed editions of Sifre Deuteronomy and Kohelet Rabba.

Key Insights and Ideas

  • Orthography is Key: The paper strongly emphasizes that the unique and often inconsistent orthography of Hebrew manuscripts is the primary barrier for existing BERT models, even those trained on historical texts.
  • Specialized Tokens Matter: The introduction of [GAP] and [ONEGAP] tokens directly addresses the problem of missing text, providing the model with explicit signals about lacunae.
  • Multi-Stage Training is Effective: The phased training approach, culminating in a manuscript-only fine-tuning phase, is shown to be critical for achieving superior performance on manuscript-specific tasks.
  • Manuscript-Specific Models Unlock New Capabilities: MsBERT's superior performance not only aids in word prediction but also demonstrates its enhanced ability to understand the context and nuances of manuscript texts, enabling tasks like citation identification.

Key Results

MsBERT significantly outperforms existing Hebrew BERT models (AlephBERT, AlephBERTGimmel, and BEREL) on two key tasks:

  • Word Prediction: On manuscript transcriptions of Shir Hashirim Rabba, MsBERT-Full achieved top-1 accuracy of 59.99%, compared to BEREL's 47.33%. Similar gains were observed on the Me'en Shiltot manuscript (MsBERT-Full: 63.99% top-1 accuracy vs. BEREL: 56.24%). The paper highlights that the substantial gap between BEREL and MsBERT underscores the importance of the manuscript-specific training data.
  • Citation Identification: MsBERT demonstrated superior recall in identifying quoted citations within texts, achieving 89.93% recall compared to BEREL's 81.94%. This indicates MsBERT's better ability to discern source material interwoven with commentary.

Limitations and Future Work

  • Liturgical Poetry: The authors acknowledge that the current MsBERT model is not expected to perform well on manuscripts containing Hebrew liturgical poetry, as this genre uses highly specialized and unique vocabulary not covered in the training data. A separate model would be needed for this.
  • Normalization Approach: While MsBERT was trained from scratch on raw manuscript text, the authors mention that future work could explore preprocessing normalization approaches and comparing their results to the current method.
  • Interactive Website: The paper mentions the release of an interactive website to allow scholars to leverage MsBERT, suggesting a practical application and a path for broader adoption.
  • HTR Systems: The authors posit that MsBERT could serve as a strong foundation for building Handwritten Text Recognition (HTR) systems for Hebrew manuscripts.

Summary updated

BibTeX
@inproceedings{shmidman2024msbert,
  author = {Avi Shmidman and Ometz Shmidman and Hillel Gershuni and Moshe Koppel},
  title = {{MsBERT: A New Model for the Reconstruction of Lacunae in Hebrew Manuscripts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure