A Distributional and Orthographic Aggregation Model for English Derivational Morphology

Daniel Deutsch, John Hewitt, Dan RothACL 2018 · OpenReview

Introduction

Derivational morphology is the linguistic process of forming new words from a root (e.g., happy → happiness). While this seems intuitive to humans, it presents significant challenges for computational models. Two primary hurdles exist: suffix ambiguity, where multiple suffixes might be plausible for a given transformation (e.g., for ground, should it be grounding or groundment?), and orthographic irregularity, where spelling changes are unpredictable (e.g., speak → speech).

Purely character-based sequence-to-sequence (seq2seq) models often struggle with these irregularities because they lack the semantic "common sense" to navigate non-predictable spelling patterns. This paper proposes a hybrid approach that bridges the gap between meaning and spelling by aggregating orthographic patterns with distributional semantics.

The Aggregation Model

The core contribution is the Aggregation Model (AGGR), which fuses two distinct modeling paradigms:

  1. The Orthographic Model (SEQ): A character-level seq2seq model using a bidirectional LSTM with attention. It excels at capturing regular orthographic patterns, such as the standard addition of "-ly" to form adverbs.
  2. The Distributional Model (DIST): A novel feed-forward network operating in word embedding space. It learns to map the vector of a root word to the vector of its derived form. By leveraging semantic embeddings, this model acts as a "semantic safety net" for irregular forms where spelling changes drastically but the semantic relationship remains consistent.
Figure 1: Architecture of the Aggregation Model, showing the integration of SEQ and DIST components into a unified hypothesis generator.

The final hypothesis is selected by a logistic regression classifier that decides which model to trust for a given input. The system is further enhanced by Dictionary Constraints (+DICT), which restrict the decoder to valid English words found in a large corpus, and Word Frequency Rescoring (+FREQ), which uses N-gram data to favor common words when the model is uncertain.

Experimental Setup

The models were evaluated using a dataset derived from NomBank, containing 4,222 training triples (root, transformation, derived word) across categories such as ADVERB, AGENT, and NOMINAL. The distributional model utilized pre-trained Google News Word2Vec vectors (929k words), while the dictionary and frequency components were informed by the Google Books Ngram corpus, filtered to a dictionary of approximately 360k words.

Results and Analysis

The aggregation model demonstrated superior performance over previous state-of-the-art (SOTA) systems across all metrics. In open-vocabulary settings, the AGGR model achieved 78.0% accuracy, a significant jump from the previous SOTA of 71.7%. When augmented with dictionary and frequency constraints (AGGR+FREQ+DICT), accuracy rose to 82.4%.

Figure 5: Performance comparison of various model configurations against the previous state-of-the-art.

A key insight is the complementary nature of the components: SEQ handles high-frequency regular suffixes, while DIST improves performance on rare or irregular forms. The most dramatic gains were seen in NOMINAL transformations (e.g., pessimistic → pessimism), where the model achieved a 49% relative error reduction. This category is particularly difficult due to the high number of competing suffixes like -ity, -ness, and -ence.

Conclusion

By combining character-level sequence modeling with distributional semantics and corpus-based constraints, this research significantly advances the state-of-the-art in English derivational morphology. While the model is currently limited by its reliance on a pre-defined embedding dictionary and cannot easily generate neologisms, its success in handling complex orthographic jumps marks a major step forward in bridging the gap between meaning and spelling.

Summary updated

BibTeX
@inproceedings{deutsch2018distributional,
  author = {Daniel Deutsch and John Hewitt and Dan Roth},
  title = {{A Distributional and Orthographic Aggregation Model for English Derivational Morphology}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=rkbOKneuWS},
}
Figure