Beyond Characters: Subword-level Morpheme Segmentation

Ben Peters, Andre MartinsSIGMORPHON 2022 · OpenReview

Abstract

This paper presents DeepSPIN’s submissions to the SIGMORPHON 2022 Shared Task on Morpheme Segmentation. We make three submissions, all to the word-level subtask. First, we show that entmax-based sparse sequence-to-sequence models deliver large improvements over conventional softmax-based models, echoing results from other tasks. Then, we challenge the assumption that models for morphological tasks should be trained at the character level by building a transformer that generates morphemes as sequences of unigram language model-induced subwords. This subword transformer outperforms all of our character-level models and wins the word-level subtask. Although we do not submit an official submission to the sentence-level subtask, we show that this subword-based approach is highly effective there as well.

What Problem Does This Paper Address?

Morpheme segmentation is the process of breaking a word into its smallest meaningful units. For example, the word "sausagemakers" can be decomposed into the root "sausage," the verb root "make," the agentive suffix "er," and the plural suffix "s."

Morpheme segmentation example for 'sausagemakers'.

Historically, neural models for this task have operated almost exclusively at the character level. The logic was that characters provide a fine-grained resolution where input and output usually align closely. However, this approach ignores the fact that languages often reuse large, stable substrings like roots and affixes. Character-level models must "re-learn" these common blocks every time they appear. This paper investigates whether subword-level modeling—common in Large Language Models but often dismissed by linguists as statistically rather than morphologically driven—can actually improve morphological segmentation accuracy.

Methods

The authors submitted three distinct models to the SIGMORPHON 2022 Shared Task, evolving from standard baselines to a novel subword-based architecture:

  1. DeepSPIN-1: A standard RNN-based sequence-to-sequence model (LSTM) using traditional softmax.
  2. DeepSPIN-2: A character-level model that replaces softmax with 1.5-entmax. Unlike softmax, which assigns a non-zero probability to every possible outcome, entmax can assign exactly zero probability to unlikely candidates, creating "sparse" distributions that lead to better calibration.
  3. DeepSPIN-3 (The Winner): A Transformer model operating on subwords induced via a Unigram Language Model (ULM). To improve robustness, the authors employed subword regularization, sampling different possible subword tokenizations during training. Like DeepSPIN-2, this model utilized entmax to handle the output distribution.

Key Insights and Ideas

Subwords vs. Characters

The most significant finding is that subword-level Transformers outperformed character-level ones. This suggests that the inductive bias of character-level modeling might be a hindrance compared to looking at larger, frequent chunks of text.

Search Complexity and Sparsity

Interestingly, subword models exhibit different search dynamics than character models. While a character model might assign high probability to a single path, the subword model often spreads probability across multiple candidates. However, the use of beam search allows the subword model to find a more accurate global solution. Across all architectures, models using entmax outperformed those using standard softmax by allowing the model to rule out impossible segments entirely.

Probability mass captured by beam search for character-level (DeepSPIN-2) vs. subword-level (DeepSPIN-3) models.

Key Results

  • Word-Level Subtask: DeepSPIN-3 won the competition with an average F-measure of 97.29 across 9 languages, outperforming the next best systems (96.85). It achieved the top result in 7 out of 9 languages.
  • Sentence-Level Success: In unofficial tests, the subword approach (DeepSPIN-sent) outperformed the official task winners in all three tested languages (Czech, English, and Mongolian), reaching an average F-measure of 91.69 compared to the winner's 89.77.

Limitations and Future Work

The authors noted that the choice of subword vocabulary size (8000) was heuristic and should be explored further, particularly for morphologically rich versus poor languages. Additionally, while the subword Transformer was successful, character-level Transformers performed poorly compared to character-level RNNs, a phenomenon that warrants future investigation. Finally, the sentence-level model did not yet incorporate surrounding context, which remains a promising avenue for further performance gains.

Summary updated

Abstract

This paper presents DeepSPIN’s submissions to the SIGMORPHON 2022 Shared Task on Morpheme Segmentation. We make three submissions, all to the word-level subtask. First, we show that entmax-based sparse sequence-to-sequence models deliver large improvements over conventional softmax-based models, echoing results from other tasks. Then, we challenge the assumption that models for morphological tasks should be trained at the character level by building a transformer that generates morphemes as sequences of unigram language model-induced subwords. This subword transformer outperforms all of our character-level models and wins the word-level subtask. Although we do not submit an official submission to the sentence-level subtask, we show that this subword-based approach is highly effective there as well.
BibTeX
@inproceedings{peters2022beyond,
  author = {Ben Peters and Andre Martins},
  title = {{Beyond Characters: Subword-level Morpheme Segmentation}},
  booktitle = {SIGMORPHON},
  year = {2022},
  url = {https://openreview.net/forum?id=OTZMvyXeIo},
}
Figure