Breaking Character: Are Subwords Good Enough for MRLs After All?

Omri Keren, Tal Avinari, Reut Tsarfaty, Omer LevyarXiv 2022 · DOI · arXiv

Introduction

In the landscape of Natural Language Processing, subword tokenization methods like Byte Pair Encoding (BPE) have become the standard for Large Language Models (LLMs). However, for Morphologically-Rich Languages (MRLs) such as Hebrew, Arabic, and Turkish, this approach has faced significant criticism. Critics argue that subwords are "shallow" statistical units that fail to capture the complex internal structures—roots, prefixes, and suffixes—inherent in these languages. This paper investigates whether a return to character-level modeling can bridge this gap.

The TavBERT Approach

To test the efficacy of character-level modeling, researchers developed TavBERT, a BERT-style Transformer encoder trained purely on characters. Unlike traditional models with vocabularies of ~50,000 subwords, TavBERT operates on a vocabulary of just a few hundred characters.

TavBERT architecture using character-level span masking to reconstruct morphological units.

A key innovation in TavBERT is its Span Masking objective. Instead of masking individual characters, the model masks random spans (averaging 5 characters). This forces the Transformer to utilize broader context to reconstruct meaningful morphological units, effectively teaching it the "grammar" of word construction from the ground up.

Experimental Setup

The models were pretrained on the OSCAR corpus for Hebrew (9.8GB), Turkish (27GB), and Arabic (32GB). Evaluation spanned two distinct categories:

  1. Morpho-syntactic Tasks: Using Universal Dependencies (UD) and SPMRL datasets for Part-of-Speech (POS) tagging and morphological disambiguation.
  2. Semantic Tasks: Including Named Entity Recognition (NER) and Question Answering (QA) using benchmarks like ParaShoot (Hebrew), TQuAD (Turkish), and TyDi QA (Arabic).

Results: The Morphological Win

TavBERT demonstrated a clear advantage in capturing surface-level grammar. In POS tagging tasks, it achieved state-of-the-art or highly competitive results. For instance, in Hebrew, TavBERT reached an F1 score of 97.15, surpassing the subword-based AlephBERT. In Arabic and Hebrew morphological disambiguation, TavBERT reduced errors by 25-39% compared to multilingual BERT (mBERT). These results suggest that character-level modeling is indeed superior for understanding "how words are built."

Results: The Semantic Wall

Despite its grammatical prowess, TavBERT encountered what the researchers call a "Semantic Wall." When tasked with high-level understanding, such as Question Answering, the performance dropped precipitously compared to subword-based models.

Comparison of TavBERT against subword models on semantic tasks (QA and NER), showing a significant performance gap.

As shown in the results, subword models like BERTurk and AraBERT significantly outperformed TavBERT in QA. In Turkish, BERTurk achieved 78.2 F1 while TavBERT trailed at 61.7 F1. A similar trend was observed in Arabic, where AraBERT reached 83.5 F1 against TavBERT’s 60.0 F1. This suggests that subwords provide a better inductive bias for semantics, likely because they often align with meaningful conceptual units that individual characters lack.

Conclusion

The study concludes that while character-level models excel at fine-grained grammatical tasks, subword tokenization remains the superior choice for semantic understanding. The "messiness" of subwords—even when they don't align perfectly with linguistic morphemes—does not prevent Transformers from learning deep semantic patterns. Future research may look toward hybrid architectures that combine the morphological sensitivity of characters with the semantic efficiency of subwords.

Summary updated

Abstract

Large pretrained language models (PLMs) typically tokenize the input string into contiguous subwords before any pretraining or inference. However, previous studies have claimed that this form of subword tokenization is inadequate for processing morphologically-rich languages (MRLs). We revisit this hypothesis by pretraining a BERT-style masked language model over character sequences instead of word-pieces. We compare the resulting model, dubbed TavBERT, against contemporary PLMs based on subwords for three highly complex and ambiguous MRLs (Hebrew, Turkish, and Arabic), testing them on both morphological and semantic tasks. Our results show, for all tested languages, that while TavBERT obtains mild improvements on surface-level tasks à la POS tagging and full morphological disambiguation, subword-based PLMs achieve significantly higher performance on semantic tasks, such as named entity recognition and extractive question answering. These results showcase and (re)confirm the potential of subword tokenization as a reasonable modeling assumption for many languages, including MRLs.
BibTeX
@misc{keren2022breaking,
  author = {Omri Keren and Tal Avinari and Reut Tsarfaty and Omer Levy},
  title = {{Breaking Character: Are Subwords Good Enough for MRLs After All?}},
  year = {2022},
  eprint = {2204.04748},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.04748},
}
Figure