Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

Miguel Ballesteros, Chris Dyer, Noah A. Smith

Abstract

We present extensions to a continuous-state dependency parsing method that makes it applicable to morphologically rich languages. Starting with a high-performance transition-based parser that uses long short-term memory (LSTM) recurrent neural networks to learn representations of the parser state, we replace lookup-based word representations with representations constructed from the orthographic representations of the words, also using LSTMs. This allows statistical sharing across word forms that are similar on the surface. Experiments for morphologically rich languages show that the parsing model benefits from incorporating the character-based encodings of words.

Introduction

In traditional natural language processing, words are often treated as "opaque" atoms—distinct units with no internal structure. While this abstraction is functional for languages with limited morphology like English, it is insufficient for morphologically rich languages (MRLs) such as Turkish, Hungarian, or Korean. In these languages, a single word can contain a root plus several suffixes indicating tense, case, or gender.

Treating these complex forms as atomic units leads to two primary issues:

  1. Sparsity: The sheer number of possible word forms means many appear only once or twice in a dataset, making it difficult for models to learn their grammatical roles.
  2. Out-of-Vocabulary (OOV) words: When a parser encounters a word not seen during training, it typically defaults to a generic "UNK" (unknown) token, discarding all specific morphological information.

Methodology

The authors build upon the Stack LSTM parser, which uses Recurrent Neural Networks to maintain the state of a transition-based parser. The key innovation is the introduction of Character-Based Word Representations.

Character-based word representation using a bidirectional LSTM to process individual characters.

Instead of retrieving a pre-defined vector from a lookup table, the model employs a bidirectional LSTM to read the sequence of characters in a word (e.g., 'p', 'l', 'a', 'y', 'i', 'n', 'g'). The final hidden states of the forward and backward passes are concatenated to form the word's vector. This allows the model to capture internal structure and share statistical information across words with similar affixes. Additionally, the authors integrated a SWAP operation into the transition system to handle non-projective trees, which are common in many MRLs.

Experiments and Results

The model was evaluated on a diverse set of languages, including the SPMRL datasets (Arabic, Basque, French, German, Hebrew, Hungarian, Korean, Polish, and Swedish), Turkish (CoNLL-X), and standard English and Chinese benchmarks.

The character-based model (Chars) consistently outperformed the word-lookup model (Words), with the most dramatic gains observed in agglutinative languages. In Korean, for example, the Unlabeled Attachment Score (UAS) rose from 78.98% to 88.98%—a 10-point improvement.

Correlation between OOV rate and the performance gain (LAS difference) of the character-based model across various languages.

A critical finding is the model's robustness to unknown words. As shown in the figure above, the performance benefit of character modeling scales with the OOV rate of the language. Because the model represents words via characters, it can generate a meaningful vector for any sequence of characters, effectively "guessing" the role of an unknown word based on its prefixes or suffixes. In Korean, replacing character-based OOV handling with a generic "UNK" token resulted in a 15.5-point drop in Labeled Attachment Score (LAS).

Discussion and Conclusion

The results suggest that "morphology is orthography" for the purposes of this model; the LSTM naturally learns that specific character sequences correlate with syntactic functions, often making explicit Part-of-Speech (POS) tags redundant.

While the character-based approach is computationally more intensive—increasing processing time from 44ms to 130ms per sentence—it provides a significant leap in accuracy for morphologically complex languages. By moving beyond atomic word lookups, the parser achieves state-of-the-art results without requiring hand-engineered morphological analyzers.

Summary updated

Abstract

We present extensions to a continuousstate dependency parsing method that makes it applicable to morphologically rich languages. Starting with a highperformance transition-based parser that uses long short-term memory (LSTM) recurrent neural networks to learn representations of the parser state, we replace lookup-based word representations with representations constructed from the orthographic representations of the words, also using LSTMs. This allows statistical sharing across word forms that are similar on the surface. Experiments for morphologically rich languages show that the parsing model benefits from incorporating the character-based encodings of words.
BibTeX
@inproceedings{ballesteros2015improved,
  author = {Miguel Ballesteros and Chris Dyer and Noah A. Smith},
  title = {{Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=By4MSffOZH},
}
Figure