A Formidable Ability: Detecting Adjectival Extremeness with DSMs

Farhan Samir, Barend Beekhuizen, Suzanne StevensonACL 2021

TL;DR

This paper introduces a novel method for identifying "extreme" adjectives (like "brilliant" or "freezing") by leveraging distributional semantic models (DSMs). They demonstrate that DSMs can capture this abstract, cross-domain property of extremeness, outperforming traditional linguistic tests and even showing promise in predicting how adverbs evolve into intensifiers.

What Problem Does This Paper Address?

Distributional semantic models (DSMs) are great at capturing word similarity within a specific domain (e.g., words related to "temperature"). However, it's less clear if they can grasp abstract semantic properties that span across multiple domains. For instance, the concept of "extremeness" applies to adjectives like "brilliant" (intelligence), "freezing" (temperature), and "colossal" (size). Humans intuitively understand this shared property, but can computational models? This paper investigates whether DSMs can learn and utilize this cross-domain abstract property, specifically focusing on "extreme adjectives" as a test case. They also explore if this ability can be applied to predict linguistic phenomena like the emergence of intensifier adverbs.

Methods

The core of the paper's methodology revolves around using DSMs, specifically word embeddings (like word2vec), to represent words.

  • Extremeness Classification:
    • DSM Centroid Method: They create a "prototype" vector by averaging the embeddings of known extreme adjectives. New adjectives are then classified based on their similarity to this centroid. This method aims to capture the abstract, cross-domain property of extremeness.
  • Distributional Test Method: This serves as a baseline, using linguistically-derived tests (e.g., how frequently an adjective co-occurs with intensifiers like "absolutely" or avoids modifiers like "very") and converting these into features for a classifier.
  • Intensifier Prediction:
    • They hypothesize that adverbs derived from extreme adjectives are more likely to become intensifiers.
  • They use a historical prediction task, training models on corpus data from decades before an adverb gained intensifier meaning to avoid circularity.
  • Two features are used:
  • EXTREMENESS: The DSM centroid measure adapted to predict whether an adverb's adjectival base is extreme.
  • SIMADJMOD: A measure of the similarity between an adverb and the adjectives it modifies, adapted from prior work.

Datasets

  • Extreme Adjectives Dataset: A dataset of 68 extreme adjectives was collected from existing literature and human-annotated datasets, covering diverse scales like desirability, intelligence, and size.
  • Non-Extreme Adjectives Dataset: 68 non-extreme adjectives were matched for frequency with the extreme adjectives from the COCA corpus.
  • Historical Corpus Data: The Google N-grams corpus (years 1800-1999) was used for historical analysis, along with diachronic word embeddings.
  • Intensifier Dataset: A set of 69 intensifiers with first attestation dates after 1800 was compiled, with 45 from a previous study. 52 intensifiers and 58 control adverbs were used in the main analysis due to missing embeddings.
  • Supplementary Analysis Datasets: For supplementary analyses, they used a dataset of 185 intensifiers and 152 control adverbs from Luo et al. (2019) with available word embeddings.

Key Insights and Ideas

  • DSMs Capture Abstract, Cross-Domain Properties: The most significant finding is that DSMs, through their similarity structure, can indeed capture abstract semantic properties like "extremeness" that transcend individual semantic domains. This is a crucial step towards models that better reflect human semantic knowledge.
  • DSM Centroid Outperforms Linguistic Tests: The DSM centroid method significantly outperformed traditional, linguistically-derived distributional tests for classifying extreme adjectives. This suggests that the rich contextual information in DSMs is more powerful than hand-crafted features.
  • Extremeness Predicts Intensifier Emergence: The paper provides strong evidence that the abstract property of extremeness is a key driver for adverbs to develop intensifying meanings. This offers a more nuanced explanation than solely relying on semantic bleaching or within-domain similarity.
  • Polysemy is a Challenge: The study highlights that polysemy (words having multiple meanings) can be a challenge for DSM-based approaches, as the model might misclassify an adjective if its dominant usage in the corpus isn't the "extreme" sense.

Key Results

  • Adjective Classification: The DSM CENTROID method achieved an AUC of 0.97 for classifying extreme adjectives, significantly outperforming the distributional test method (AUC = 0.78) and random chance (AUC = 0.50).
  • Intensifier Prediction: The EXTREMENESS feature achieved an AUC of 0.86 in predicting intensifier emergence, outperforming the SIMADJMOD feature (AUC = 0.61). This held even when controlling for data sparsity.
  • Supplementary Analysis: Even in a more controlled setting designed to minimize domain-specific confounds, the DSM CENTROID method still robustly outperformed the distributional test method. In the supplementary intensifier prediction task, the EXTREMENESS feature also showed strong performance (AUC = 0.90 vs. 0.84 for SIMADJMOD).

Limitations and Future Work

  • Polysemy: The authors acknowledge that their approach relies on the dominant sense of a word in the corpus. For polysemous words, this can lead to misclassification if the "extreme" sense isn't the most frequent. Future work could explore sense-annotated corpora or methods to disentangle senses.
  • Data Sparsity: In the historical analysis, data sparsity (especially for older adverbs and embeddings) was a significant challenge, requiring adaptations to the methods.
  • Future Work: The authors suggest exploring influence functions for NLP to better understand how DSMs represent abstract properties. They also propose investigating the role of semantic bleaching in more detail and applying their methods to other abstract semantic classes.

Summary updated

BibTeX
@inproceedings{samir2021formidable,
  author = {Farhan Samir and Barend Beekhuizen and Suzanne Stevenson},
  title = {{A Formidable Ability: Detecting Adjectival Extremeness with DSMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure