From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?

Reut Tsarfaty, Dan Bareket, Stav Klein, Amit Seker

TL;DR

Standard neural NLP models assume an English-like linear word structure, causing them to struggle when applied to Morphologically-Rich Languages (MRLs) like Hebrew, Arabic, or Turkish. This paper argues that the key to unlocking high performance in neural MRL processing lies in resurrecting pre-neural symbolic strategies—specifically, joint lattice-based modeling. The authors demonstrate this with a novel COPYNET architecture that integrates symbolic morphological lattices into a neural pipeline, nearly closing the performance gap with classical state-of-the-art parsers.

What Problem Does This Paper Address?

Deep learning has driven massive breakthroughs in NLP, but these advances have been heavily optimized for English—a language with a rigid word order and relatively simple word structures. In contrast, Morphologically-Rich Languages (MRLs) pack significant syntactic information (like prepositions, pronouns, gender, and tense) directly into single, highly complex word forms.

This structural density creates three core challenges:

  1. Architectural: Standard tokenizers (like WordPieces in BERT) split words based on character frequencies rather than actual linguistic morphemes, failing to capture non-concatenative grammar.
  2. Modeling: A single word form can have multiple, highly ambiguous morphological interpretations. Standard neural models commit to a single word vector before analyzing the global context, leading to cascading errors.
  3. Lexical: The sheer number of possible word mutations leads to extreme data sparseness and massive out-of-vocabulary (OOV) issues.

The authors address the widening performance gap between English and MRLs in the deep learning era, arguing that purely end-to-end neural models cannot implicitly "learn" these intricate morphological structures from raw text alone.

Methods

The authors propose a paradigm shift called Neural Models for MRLs (NMRL), which advocates for blending symbolic linguistic knowledge with deep learning architectures.

To evaluate this approach, they design a Seq2Seq COPYNET architecture for a Hebrew multi-tagging task (predicting the sequence of Part-of-Speech tags for all morphemes inside a raw token). Instead of feeding raw text or standard subword tokens into the network, the COPYNET takes a morphological lattice as input. This lattice is a directed graph generated by a traditional, rule-based lexicon that represents all possible morphological breakdowns of a word. The COPYNET uses a copy-attention mechanism to jointly select the correct morphological segments and their corresponding tags directly from the lattice, allowing the model to delay disambiguation until global sentence context is processed.

This approach was benchmarked against standard neural baselines:

  • A language-agnostic LSTM-CRF (tested with character embeddings and morphologically-trained FastText embeddings).
  • A fine-tuned multilingual BERT model.
  • A pre-neural state-of-the-art (SOTA) morpho-syntactic parser (More et al., 2019).

Datasets

  • SPMRL Hebrew Treebank: The standard Hebrew section from the Statistical Parsing of Morphologically-Rich Languages (SPMRL) shared task, utilizing the standard split of 5,000 sentences for training and 500 sentences for evaluation.
  • Adler and Elhadad Lexicon (2006): A wide-coverage external Hebrew lexicon used to generate the morphological input lattices for the COPYNET and pre-neural SOTA models.

Key Insights and Ideas

  • The "End-to-End" Illusion: The assumption that deep neural networks will automatically learn all relevant linguistic features from raw text does not hold true for complex, high-fusion languages like Hebrew.
  • Oracle vs. Reality: Standard neural models (like BERT and LSTMs) perform exceptionally well if they are given perfect, human-annotated word segmentations ("Oracle Segmentation"). However, when forced to process raw tokens or automatically predicted segments, their performance plummets dramatically.
  • Lattices as a Neural Input: Morphological lattices successfully preserve ambiguity. By feeding the entire graph of possibilities into the neural network, the model avoids making premature errors during tokenization.

Key Results

The models were evaluated using F1-scores on joint Segmentation and POS tagging (Seg+POS):

  • Pre-Neural SOTA: Achieved the highest baseline score of 95.5 F1 using explicit symbolic components.
  • Standard LSTM-CRF (Raw Tokens): Performed poorly, scoring only 73.38 F1, proving that standard sequential models fail on raw MRL text.
  • Multilingual BERT (Raw Tokens): Reached 92.57 F1, showing strong contextual capabilities but still lagging behind the pre-neural baseline.
  • Seq2Seq COPYNET (Proposed NMRL Approach): Achieved 95.1 F1 using raw lattices. This successfully closed the gap with the pre-neural SOTA without requiring gold-standard segmentations.

Limitations and Future Work

The primary limitation highlighted is architectural incompatibility. Morphological lattices represent a partial order of morphemes, whereas dominant neural architectures like LSTMs and Transformers (BERT) strictly require a linear sequence of inputs.

The authors note that finding a way to seamlessly integrate contextualized word embeddings (like BERT) into joint, lattice-based models remains an open and fascinating challenge. Additionally, this preliminary study was restricted to a single task (multi-tagging) and one language (Hebrew); future work must expand these strategies to syntactic parsing, semantic analysis, and other MRLs like Arabic or Turkish.

Summary updated

Abstract

It has been exactly a decade since the first establishment of SPMRL, a research initiative unifying multiple research efforts to address the peculiar challenges of Statistical Parsing for Morphologically-Rich Languages (MRLs).Here we reflect on parsing MRLs in that decade, highlight the solutions and lessons learned for the architectural, modeling and lexical challenges in the pre-neural era, and argue that similar challenges re-emerge in neural architectures for MRLs. We then aim to offer a climax, suggesting that incorporating symbolic ideas proposed in SPMRL terms into nowadays neural architectures has the potential to push NLP for MRLs to a new level. We sketch strategies for designing Neural Models for MRLs (NMRL), and showcase preliminary support for these strategies via investigating the task of multi-tagging in Hebrew, a morphologically-rich, high-fusion, language
BibTeX
@inproceedings{tsarfaty2020from,
  author = {Reut Tsarfaty and Dan Bareket and Stav Klein and Amit Seker},
  title = {{From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.660},
}
Figure