Abstract
While the success of pre-trained language models has largely eliminated the need for high-quality static word vectors in many NLP applications, such vectors continue to play an important role in tasks where words need to be modelled in the absence of linguistic context. In this paper, we explore how the contextualised embeddings predicted by BERT can be used to produce high-quality word vectors for such domains, in particular related to knowledge base completion, where our focus is on capturing the semantic properties of nouns. We find that a simple strategy of averaging the contextualised embeddings of masked word mentions leads to vectors that outperform the static word vectors learned by BERT, as well as those from standard word embedding models, in property induction tasks. We notice in particular that masking target words is critical to achieve this strong performance, as the resulting vectors focus less on idiosyncratic properties and more on general semantic properties. Inspired by this view, we propose a filtering strategy which is aimed at removing the most idiosyncratic mention vectors, allowing us to obtain further performance gains in property induction.
Introduction
The advent of contextualized embeddings, such as those from BERT and RoBERTa, has revolutionized natural language processing by enabling word representations to adapt to their linguistic environment. Despite this, static word vectors—fixed representations for individual words—remain critical for applications where context is unavailable, including information retrieval, zero-shot learning, and knowledge base completion.
Traditional static embeddings like GloVe or Word2Vec are often limited to capturing distributional co-occurrence. While Large Language Models (LLMs) encode significantly richer semantic knowledge, extracting a single, high-quality vector that represents the "prototypical" properties of a noun is non-trivial. Simple averaging of contextual embeddings often incorporates "noise" from idiosyncratic or highly specific sentence contexts. This paper proposes a methodology to distill the deep semantic knowledge of LLMs into superior static representations for nouns.
Methodology
The authors introduce a two-step process to derive refined static vectors from contextualized models.
Masked Mention Vectors
The core innovation lies in the use of masking. Instead of extracting the embedding for a target word directly, the word is replaced with a [MASK] token. The model is then tasked with predicting the vector for this mask based on the surrounding context. This approach effectively treats the sentence as a "bag of properties," forcing the model to infer semantic attributes (e.g., "edible," "yellow") from the context rather than the word's identity.
Selective Filtering
To address the issue of idiosyncratic contexts—sentences that describe rare or non-prototypical situations—a selective filtering strategy is employed. Using a $k$-nearest neighbor ($k$-NN) approach in the vector space, the authors identify mention vectors that are outliers or overly specific. By removing these "filtered out" mentions, the resulting average focuses on the general semantic properties of the noun.
Experimental Results
The proposed method, AVGfilt, was evaluated against standard baselines (GloVe, Skip-gram, and standard BERT averaging) across benchmarks including X-McRae, CSLB, WordNet, and BabelNet.
Property Induction and Ontology Completion
AVGfilt consistently outperformed all baselines in property induction tasks. For instance, on the Morrow taxonomic dataset, it achieved an F1 score of 70.1%, compared to 61.2% for Skip-gram. The method also proved highly effective in ontology completion, demonstrating its utility for refining structured knowledge bases.
Layer-Wise Performance
Analysis of BERT's internal representations revealed that the most useful semantic information for nouns is often found in middle-to-late layers rather than the final layer. As shown in the figure below, performance on the WordNet dataset peaks around layers 8-12, suggesting that these layers best capture the general semantic properties required for property induction.
Discussion and Conclusion
The study highlights a "similarity paradox": while the masking and filtering approach is superior for identifying semantic properties, it performs less effectively on word similarity tasks. This suggests that human similarity judgments are influenced more by relatedness and co-occurrence than by strict property overlap.
While currently focused on nouns and requiring significant computational resources to process contexts, this research provides a robust framework for extracting prototypical concept representations from LLMs. Future work will explore extending this strategy to other parts of speech and more complex reasoning tasks.
Summary updated