Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings

Na Li, Zied Bouraoui, José Camacho-Collados, Luis Espinosa Anke, Qing Gu, Steven Schockaert

TL;DR

Researchers have developed a clever way to "distill" the deep contextual knowledge of Large Language Models (LLMs) like BERT into high-quality static word vectors. By masking a noun in various sentences and filtering out overly specific or "weird" contexts, they created embeddings that are significantly better at capturing the general semantic properties of objects—like knowing that a "banana" is edible, yellow, and a fruit.

What Problem Does This Paper Address?

While contextualized embeddings (where the word "bank" changes meaning depending on whether you're talking about money or a river) have revolutionized NLP, static word vectors (one fixed vector per word) are still essential. They are used in information retrieval, zero-shot learning, and building knowledge bases where words often appear without any surrounding context.

Standard static vectors like GloVe or Word2Vec are "shallow"—they mostly capture how often words appear near each other. LLMs like BERT contain much richer knowledge, but simply averaging a word's various contextual embeddings often results in "noisy" vectors that focus on specific, quirky sentences rather than the general properties of the noun itself. The authors set out to bridge this gap: how do we get a "goldilocks" vector that represents the essence of a noun?

Methods

The authors propose a two-step strategy to extract better static representations from BERT and RoBERTa:

  1. Masked Mention Vectors: Instead of taking the embedding of the word "banana" in a sentence, they replace "banana" with a [MASK] token. They then use the vector BERT predicts for that mask. This forces the model to rely entirely on the context to describe the word. Essentially, the sentence becomes a "bag of properties" (e.g., "A [MASK] is a yellow fruit" reveals the properties yellow and fruit).
  2. Selective Filtering: Not all sentences are equally useful. Some are "idiosyncratic"—they describe a very specific or rare situation (e.g., "The banana equivalent dose is a measure of radiation"). To remove these, the authors use a $k$-nearest neighbor ($k$-NN) approach. If a specific mention vector is surrounded only by other mentions of the same word in the vector space, it’s likely too specific/idiosyncratic and is discarded. They then average the remaining "general" vectors to create the final static embedding.

Datasets

The authors evaluated their method across several diverse benchmarks:

  • Commonsense Properties: X-McRae and CSLB (e.g., is a "table" made of wood?).
  • Taxonomic Categories: Morrow and WordNet supersenses (e.g., is a "dog" an animal?).
  • Topical Domains: BabelNet domains (e.g., is "football" related to sports?).
  • Word Similarity: SimLex-999, WordSim-353, and SemEval-2017.
  • Ontology Completion: Domain-specific datasets like Wine, Economy, and Olympics.

Key Insights and Ideas

  • Masking as a Feature Extractor: The "aha moment" is that masking the target word transforms the LLM from a word-recognizer into a property-inferer. It forces the model to represent what the context reveals about the word, rather than just the word's identity.
  • General vs. Idiosyncratic: The paper highlights that for many tasks, we don't want the most "accurate" representation of every sentence a word has ever appeared in. We want the "prototypical" representation. By filtering out the outliers, the resulting vector becomes a much stronger representation of the concept's core attributes.
  • The Similarity Paradox: Interestingly, the authors found that while their masking/filtering method is superior for identifying properties (is a whale a mammal?), it actually performs worse on word similarity tasks (is a whale like a dolphin?). This suggests that human similarity judgments rely more on co-occurrence and "relatedness" than on strict semantic properties.

Key Results

  • Property Induction: The proposed method (AVGfilt) outperformed standard word embeddings (GloVe, Skip-gram) and standard BERT averaging across all five lexical classification datasets. For example, on the Morrow taxonomic dataset, it achieved an F1 score of 70.1% compared to Skip-gram's 61.2%.
  • Ontology Completion: In the downstream task of predicting missing rules in an ontology, the filtered masked vectors consistently beat the baselines, showing their utility for building and cleaning structured knowledge bases.
  • Layer Wisdom: The authors confirmed that the "best" knowledge isn't always at the very last layer of BERT. For noun properties, middle-to-late layers (around layer 8-10 for some tasks) often provide the most useful semantic information.

Limitations and Future Work

  • Noun-Centric: The study focuses exclusively on nouns. It remains to be seen if this "mask-and-filter" strategy works as well for verbs (actions) or adjectives (attributes).
  • Computational Cost: Generating these vectors requires sampling hundreds of sentences per word and running them through a large LM, which is significantly more expensive than training a simple Skip-gram model.
  • Filtering Sensitivity: The filtering strategy relies on a $k$ parameter that needs tuning. If $k$ is too high, you might lose valuable nuances; if it's too low, you keep too much noise.

The authors suggest that future work could involve more sophisticated ways to select "prototypical" sentences or applying these high-quality vectors to more complex reasoning tasks in the Semantic Web.

Summary updated

Abstract

While the success of pre-trained language models has largely eliminated the need for high-quality static word vectors in many NLP applications, static word vectors continue to play an important role in tasks where word meaning needs to be modelled in the absence of linguistic context. In this paper, we explore how the contextualised embeddings predicted by BERT can be used to produce high-quality word vectors for such domains, in particular related to knowledge base completion, where our focus is on capturing the semantic properties of nouns. We find that a simple strategy of averaging the contextualised embeddings of masked word mentions leads to vectors that outperform the static word vectors learned by BERT, as well as those from standard word embedding models, in property induction tasks. We notice in particular that masking target words is critical to achieve this strong performance, as the resulting vectors focus less on idiosyncratic properties and more on general semantic properties. Inspired by this view, we propose a filtering strategy which is aimed at removing the most idiosyncratic mention vectors, allowing us to obtain further performance gains in property induction.
BibTeX
@inproceedings{li2021modelling,
  author = {Na Li and Zied Bouraoui and José Camacho-Collados and Luis Espinosa Anke and Qing Gu and Steven Schockaert},
  title = {{Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=VZApBRgPD_m},
}

Loading related content...

Figure