‘Indicatements’ that character language models learn English morpho-syntactic units and regularities

Yova Kementchedjhieva, Adam Lopez

TL;DR

This paper investigates whether character-level language models (LMs) can learn abstract morphological and syntactic regularities in English. By probing a character LM, the authors found that it can indeed identify word and morpheme boundaries, and even capture selectional restrictions of derivational suffixes, suggesting a surprising level of linguistic understanding.

What Problem Does This Paper Address?

Character-level language models (like those used in speech recognition and machine translation) have the advantage of handling out-of-vocabulary words by processing text character by character. While it's often claimed these models learn morphosyntactic properties, there's been a lack of empirical evidence to support this. This paper aims to answer: "What do these models really learn about morphology, and how do they learn it?" They want to understand if these models go beyond simply memorizing character sequences and actually grasp the underlying linguistic structure of words and sentences.

Methods

The researchers used a character-level Recurrent Neural Network (RNN) with LSTM units, trained on a large English Wikipedia corpus. To probe the model's understanding, they employed several techniques:

  • Generation: Observing the model's output to see if it applied morphological processes productively.
  • Hidden Unit Analysis: Analyzing specific "hidden units" within the neural network to see if they activated in response to linguistic phenomena like punctuation or morpheme boundaries.
  • Morphological Segmentation Probe: Training a separate model (C2M) that uses the character LM's learned embeddings to predict morpheme boundaries. This tests how well the LM's internal representations capture morphological structure.
  • Part-of-Speech (POS) Tagging Probe: Training another model (C2T) using the LM's embeddings to predict POS tags, assessing its understanding of syntactic categories.
  • Selectional Restriction Test: Measuring the LM's probability of suffix attachment to different base word categories (noun, verb, adjective) to see if it understands syntactic constraints on morphology.

Datasets

  • Training: The first 7 million character tokens from the English Wikipedia corpus.
  • Morphological Segmentation Evaluation: A combined set of gold standard segmentations from MorphoChallenge 2010 and Hutmegs 1.0 (2275 word forms).
  • POS Tagging Evaluation: The English UD corpus with UD POS tags.
  • Selectional Restriction Evaluation: Nonce words (generated by the LM and filtered) were used as bases, paired with suffixes.

Key Insights and Ideas

  • Morpheme Boundaries from Word Boundaries: The LM appears to learn morpheme boundaries by analogy with word boundaries. This is facilitated by English's relatively simple morphology where morpheme and word boundaries often coincide.
  • "Word Unit" as a Key Detector: A specific hidden unit in the LM strongly activates at word boundaries, suggesting it plays a role in recognizing complete linguistic units. This unit seems to anticipate following units, hinting at an understanding of sequential linguistic structure.
  • Productive Morphology: The LM can generate novel words that follow English morphological rules (e.g., adding "-ed" to a verb), indicating it's not just memorizing but applying patterns.
  • Syntactic Awareness: The model shows an understanding of syntactic categories, as evidenced by its ability to predict POS tags and its grasp of selectional restrictions for derivational suffixes.
  • Suffix -ion Anomaly: The suffix "-ion" is a prime example of the LM's learning process. While it's a deverbal suffix, the LM incorrectly favors it on nominal bases. This is attributed to the high frequency of "-ion" appearing with nominal bases in the training data, biasing the model's expectations.

Key Results

  • The character LM generates mostly real English words, with only about 1 in 20 tokens being a nonce word.
  • The morphological segmentation probe (C2M) achieved an F1 score of 53.3%, which is lower than state-of-the-art CRF models but still suggests the LM's embeddings contain useful morphological information.
  • The C2M model performed particularly well at identifying word endings (EOW), achieving a 90.90% F1 score.
  • The POS tagging probe (C2T) achieved 78.85% character-level accuracy and 87.06% word-level accuracy, indicating the LM captures significant syntactic information.
  • Eleven out of nineteen tested derivational suffixes showed the expected behavior, attaching more frequently to their correct syntactic base category, demonstrating the LM's understanding of selectional restrictions.

Limitations and Future Work

  • English-Centricity: The paper acknowledges that English's relatively simple inflectional morphology makes it an ideal candidate for character-level LMs. For languages with richer morphology, learning these regularities might be significantly harder without explicit subword segmentation.
  • Interpretability: The authors were "lucky" to find a particularly interpretable "word unit." They suggest that future work should investigate the reliability of such findings across different model architectures and initializations.
  • Further Analysis: The paper calls for analyses of other languages to understand how character-level models handle more complex morphology. It also suggests exploring how explicit morphological annotations might further improve language modeling accuracy for less inflectional languages.

Summary updated

Abstract

Character language models have access to surface morphological patterns, but it is not clear whether or how they learn abstract morphological regularities. We instrument a character language model with several probes, finding that it can develop a specific unit to identify word boundaries and, by extension, morpheme boundaries, which allows it to capture linguistic properties and regularities of these units. Our language model proves surprisingly good at identifying the selectional restrictions of English derivational morphemes, a task that requires both morphological and syntactic awareness. Thus we conclude that, when morphemes overlap extensively with the words of a language, a character language model can perform morphological abstraction.
BibTeX
@inproceedings{kementchedjhieva2018indicatements,
  author = {Yova Kementchedjhieva and Adam Lopez},
  title = {{‘Indicatements’ that character language models learn English morpho-syntactic units and regularities}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure