Learning to Predict Novel Noun-Noun Compounds

Prajit Dhar, Lonneke van der PlasACL 2019

TL;DR

This paper introduces a novel task in NLP: predicting plausible, unseen noun-noun compounds. They propose temporally and contextually aware neural models that learn from attested compounds and corrupted examples, achieving an impressive 85% accuracy in predicting compounds that appear in future data. This work is crucial for understanding and modeling linguistic creativity and building more adaptive NLP systems.

What Problem Does This Paper Address?

The paper tackles the challenge of modeling linguistic creativity, specifically focusing on how humans form new noun-noun compounds. While compounding is a fundamental way to create new concepts in language, computationally modeling this process has received limited attention. Existing work often focuses on identifying rare but existing compounds or uses static datasets. This paper addresses the gap by proposing a method to predict novel, plausible compounds that haven't yet appeared in large corpora, aiming to capture the dynamic and creative nature of language. This is important for NLP systems that need to understand and generate novel linguistic expressions.

Methods

The authors propose three models: a Distributional Feature Model (DFM), a Neural Network Model (NNM), and Distributional Semantic Models (DSMs) as a baseline.

  • Distributional Feature Model (DFM): This model uses handcrafted features derived from word co-occurrence statistics. These include Pointwise Mutual Information (PPMI), Log Likelihood-Ratio (LLR), and Local Mutual Information (LMI) to capture relationships between words. It also incorporates similarity measures between compound constituents and the compound itself. The DFM uses a gradient boosting model (XGBoost) for supervised learning.
  • Neural Network Model (NNM): This model uses a compositional neural network architecture, inspired by previous work on selectional preferences. It takes vector representations of the modifier and head as input and passes them through a feed-forward network with a ReLU activation function in the hidden layer. The output is a plausibility score.
  • Contextual and Temporal Aspects: A key innovation is the incorporation of two aspects:
  • CompoundCentric vs. CompoundAgnostic: CompoundAgnostic uses standard word embeddings, while CompoundCentric learns embeddings that are sensitive to a word's role (modifier or head) within a compound.
  • DecadeCentric vs. DecadeAgnostic: DecadeCentric models are trained on data from previous decades to predict compounds in a later decade, capturing temporal shifts in language. DecadeAgnostic models ignore temporal information.

Negative data for training is generated by corrupting attested compounds (replacing either the head or modifier with a random constituent from another compound).

Datasets

The primary dataset used is the Google Books Ngram Corpus (Michel et al., 2010), spanning five centuries. For their study, they extracted unigrams and bigrams from 20 decades (1800s-1990s) for training and the 2000s for testing. Compounds were considered "novel" if they appeared in the 2000s decade with a frequency of at least 3. The 2000s decade was excluded from training to ensure the prediction of truly unseen compounds.

Key Insights and Ideas

  • Temporal and Contextual Awareness is Crucial: The models that incorporate temporal information (DecadeCentric) significantly outperform those that don't (DecadeAgnostic), highlighting that language evolution is key to predicting new word formations. Similarly, models sensitive to a word's role in a compound (CompoundCentric) perform better.
  • Head Corruption is a Better Negative Signal: Corrupting the head of a compound to generate negative examples yields better results than corrupting the modifier. This is intuitively explained by the head's dominant role in determining the compound's meaning.
  • Corpus-based Evaluation is Pessimistic: While achieving 85% accuracy on corpus data is impressive, human evaluation reveals that this metric is pessimistic. Many compounds predicted as "incorrect" (not found in the test decade) are still deemed plausible by humans, suggesting the corpus is not yet "invented" these compounds.
  • Modeling Linguistic Creativity is Feasible: The success of the models, particularly the NNM, demonstrates that computational approaches can effectively learn and predict novel linguistic creations, opening doors for more creative NLP applications.

Key Results

The Neural Network Model (NNM), when using both CompoundCentric and DecadeCentric aspects, achieved the highest accuracy of 85% on the corpus data for the CorruptHead scenario. This means that for 85% of the generated novel compounds, the model correctly identified them as plausible by checking their presence in the unseen 2000s decade.

In general, DecadeCentric models outperformed DecadeAgnostic models, and CompoundCentric models outperformed CompoundAgnostic models across all tested models (DFM, DSM, NNM).

Human evaluation on a subset of compounds predicted as incorrect by the corpus evaluation showed that approximately 5% of these were rated as plausible (score of 3 or higher on a 0-4 scale) by annotators, indicating the corpus-based evaluation is conservative.

Limitations and Future Work

  • Limitations:
    • The corpus-based evaluation is acknowledged as pessimistic, as it only counts compounds that have already appeared in the latest decade.
  • The current experiments are limited to noun-noun (N-N) compounds.
  • Future Work:
    • Investigate the correlation between human plausibility judgments and occurrence in the latest decade to rigorously test the evaluation method.
  • Experiment with other models like cDSMs and enhanced neural network architectures (e.g., bidirectional LSTMs, attention-based networks).
  • Expand the scope to include closed compounds, hyphenated compounds, and compounds involving other parts of speech (e.g., adjective-noun compounds).

Summary updated

BibTeX
@inproceedings{dhar2019learning,
  author = {Prajit Dhar and Lonneke van der Plas},
  title = {{Learning to Predict Novel Noun-Noun Compounds}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure