Intensionality was only alleged: On adjective-noun composition in distributional semantics

Gemma Boleda, Marco Baroni, Nghia The Pham, Louise McNallyIwcs 2013

TL;DR

This paper investigates how to computationally model adjective-noun phrases using distributional semantics. They find that models inspired by higher-order predication perform best, and surprisingly, that intensional adjectives (like "alleged") are not harder to model than non-intensional ones ("white"). The key factors influencing composition quality are noun polysemy and adjective typicality, not the traditional intensionality distinction.

What Problem Does This Paper Address?

Distributional semantics has been highly successful at modeling word meanings by analyzing their co-occurrence patterns in large text corpora. However, extending this to model phrase meanings (semantic composition) is a significant challenge. Adjective-noun (AN) phrases are a particularly interesting test case because they are syntactically simple but semantically complex, with the adjective's effect on the noun varying greatly.

Traditionally, adjectives have been classified based on their semantic properties, such as being "intersective" (e.g., "white towel" implies both "white" and "towel"), "subsective" (e.g., "skillful surgeon" implies "surgeon" but not necessarily "skillful" in general), or "intensional" (e.g., "alleged murderer" doesn't imply the person is a murderer). Intensional adjectives are thought to be particularly difficult to model computationally because they don't simply describe attributes but rather modify the noun's meaning in more abstract ways. This paper aims to test whether current distributional composition models can effectively handle AN phrases, particularly those involving intensional adjectives, and to understand what factors contribute to successful composition.

Methods

The authors experiment with several compositional models for AN phrases, comparing them against "observed vectors" (phrase vectors directly extracted from the corpus). The models fall into two main categories:

  • Feature Combination Models: These models treat word vectors as sets of features and combine them through simple operations like addition (weighted additive) or element-wise multiplication (multiplicative). These are inspired by simple feature union or intersection.
  • Matrix-Based Models: These models are inspired by higher-order predication in formal semantics, where adjectives are treated as functions (represented by matrices) that transform noun vectors. Examples include the "full additive" model (using weight matrices) and the "lexical function" model (where each adjective has its own unique matrix).

The models are trained and evaluated using semantic spaces derived from large corpora. Model parameters are optimized by minimizing the distance between compositionally derived phrase vectors and their corresponding observed vectors. The quality of the models is assessed by comparing the cosine similarity between predicted and observed phrase vectors, and by analyzing the nearest neighbors of the predicted phrase vectors.

Datasets

The study uses a large corpus formed by concatenating the ukWaC corpus, Wikipedia, and the British National Corpus (approximately 2.8 billion tokens). This corpus is dependency-parsed to extract adjective-noun modification pairs.

  • Training Data: The authors use the top 10K most frequent AN phrases (excluding the very top 1,000 to avoid highly fixed expressions) for training, along with observed vectors for adjectives and nouns. For the lexical function model, which requires adjective-specific parameters, they use all available phrase vectors containing that adjective (excluding test items).
  • Evaluation Data: A specific test set of 16 intensional and 16 non-intensional adjectives is created, paired by frequency. For each adjective, a set of AN phrases with a corpus frequency of at least 20 (or down to 5 for less frequent adjectives, with a cap of 200 phrases) is used. A random sample of 50 phrases per adjective is selected for testing, totaling 1,600 phrases.

Key Insights and Ideas

  • Intensionality is Not the Key Difficulty: Contrary to theoretical expectations, the study finds no significant difference in how well models handle intensional versus non-intensional adjectives. This suggests that the traditional distinction might not be as crucial for distributional composition as previously thought.
  • Higher-Order Models Excel: Models that treat adjectives as functions (matrix-based models like the lexical function model) consistently outperform simpler feature combination models. This aligns with the idea that adjective modification is more than just a simple merging of features.
  • Noun Polysemy Matters: The more meanings a noun has (i.e., the more polysemous it is), the harder it is to accurately compose with an adjective. This is because the adjective has a larger "shifting" effect on a polysemous noun's meaning, making the composition more impactful and potentially harder to capture.
  • Typicality is Crucial: When an adjective describes a typical attribute of a noun (e.g., "ripe raspberry"), the resulting phrase vector is more similar to both the adjective and noun vectors. Conversely, less typical combinations lead to greater divergence. This suggests that typicality plays a significant role in how adjective-noun meanings interact.
  • Composition is More Than Feature Union/Intersection: The poor performance of the multiplicative model (which acts like feature intersection) and the success of matrix-based models suggest that adjective-noun composition is a more complex transformation than simple feature merging.

Key Results

  • Lexical Function Model Dominates: The lexical function model achieved the highest cosine score (0.60) between predicted and observed phrase vectors, outperforming all other models, including a simple "noun only" baseline (0.40).
  • No Difference for Intensionality: Across all models, the average cosine scores for intensional and non-intensional phrases were virtually identical, indicating no inherent difficulty for intensional adjectives.
  • Matrix Models Produce Better Neighbors: Matrix-based models produced nearest neighbors that were more often other AN phrases, aligning better with observed data, whereas feature combination models tended to produce more single-word neighbors (adjectives or nouns).
  • Polysemy and Typicality Effects: Statistical analysis confirmed that noun polysemy negatively impacts composition quality (lower cosine scores), while adjective typicality positively impacts it (higher cosine scores). These effects were found to be independent of whether the adjective was intensional or not.

Limitations and Future Work

  • Bag-of-Words Representation: The authors acknowledge that the bag-of-words representation used for dimensions might not fully capture subtle semantic differences, and suggest this could be a reason why intensionality differences weren't more apparent.
  • Focus on Entailment: The paper critiques the traditional focus on entailment relations in formal semantics, arguing it has obscured other important factors like polysemy and typicality.
  • Future Work: The authors suggest further research into how polysemy and typicality can be better reflected in semantic representations and composition processes. They also propose exploring alternative association measures for typicality and investigating the role of syntax more deeply. The pilot study on rare phrases suggests that predicted vectors might be more useful than observed vectors for less frequent AN combinations, warranting further investigation.

Summary updated

BibTeX
@inproceedings{boleda2013intensionality,
  author = {Gemma Boleda and Marco Baroni and Nghia The Pham and Louise McNally},
  title = {{Intensionality was only alleged: On adjective-noun composition in distributional semantics}},
  booktitle = {Iwcs},
  year = {2013},
}
Figure