Introduction
In the landscape of Natural Language Processing, subword tokenization methods like Byte Pair Encoding (BPE) have become the standard for Large Language Models (LLMs). However, for Morphologically-Rich Languages (MRLs) such as Hebrew, Arabic, and Turkish, this approach has faced significant criticism. Critics argue that subwords are "shallow" statistical units that fail to capture the complex internal structures—roots, prefixes, and suffixes—inherent in these languages. This paper investigates whether a return to character-level modeling can bridge this gap.
The TavBERT Approach
To test the efficacy of character-level modeling, researchers developed TavBERT, a BERT-style Transformer encoder trained purely on characters. Unlike traditional models with vocabularies of ~50,000 subwords, TavBERT operates on a vocabulary of just a few hundred characters.
A key innovation in TavBERT is its Span Masking objective. Instead of masking individual characters, the model masks random spans (averaging 5 characters). This forces the Transformer to utilize broader context to reconstruct meaningful morphological units, effectively teaching it the "grammar" of word construction from the ground up.
Experimental Setup
The models were pretrained on the OSCAR corpus for Hebrew (9.8GB), Turkish (27GB), and Arabic (32GB). Evaluation spanned two distinct categories:
- Morpho-syntactic Tasks: Using Universal Dependencies (UD) and SPMRL datasets for Part-of-Speech (POS) tagging and morphological disambiguation.
- Semantic Tasks: Including Named Entity Recognition (NER) and Question Answering (QA) using benchmarks like ParaShoot (Hebrew), TQuAD (Turkish), and TyDi QA (Arabic).
Results: The Morphological Win
TavBERT demonstrated a clear advantage in capturing surface-level grammar. In POS tagging tasks, it achieved state-of-the-art or highly competitive results. For instance, in Hebrew, TavBERT reached an F1 score of 97.15, surpassing the subword-based AlephBERT. In Arabic and Hebrew morphological disambiguation, TavBERT reduced errors by 25-39% compared to multilingual BERT (mBERT). These results suggest that character-level modeling is indeed superior for understanding "how words are built."
Results: The Semantic Wall
Despite its grammatical prowess, TavBERT encountered what the researchers call a "Semantic Wall." When tasked with high-level understanding, such as Question Answering, the performance dropped precipitously compared to subword-based models.
As shown in the results, subword models like BERTurk and AraBERT significantly outperformed TavBERT in QA. In Turkish, BERTurk achieved 78.2 F1 while TavBERT trailed at 61.7 F1. A similar trend was observed in Arabic, where AraBERT reached 83.5 F1 against TavBERT’s 60.0 F1. This suggests that subwords provide a better inductive bias for semantics, likely because they often align with meaningful conceptual units that individual characters lack.
Conclusion
The study concludes that while character-level models excel at fine-grained grammatical tasks, subword tokenization remains the superior choice for semantic understanding. The "messiness" of subwords—even when they don't align perfectly with linguistic morphemes—does not prevent Transformers from learning deep semantic patterns. Future research may look toward hybrid architectures that combine the morphological sensitivity of characters with the semantic efficiency of subwords.
Summary updated