Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models

Na Li, Hanane Kteich, Zied Bouraoui, Steven Schockaert

TL;DR

This paper introduces novel contrastive learning strategies to distill high-quality semantic concept embeddings from pre-trained language models. By fine-tuning models to make embeddings of sentences expressing similar properties closer, they achieve state-of-the-art performance on tasks like word classification and ontology completion, significantly outperforming existing methods.

What Problem Does This Paper Address?

Representing the meaning of concepts as dense vectors (embeddings) is crucial for many NLP tasks, from zero-shot learning to ontology completion. While pre-trained language models (LMs) like BERT have revolutionized NLP, their direct application to concept embeddings has yielded only modest improvements. A common approach is to average the contextualized embeddings of a concept's mentions in a corpus. However, this method is suboptimal for two key reasons: 1) contextualized word vectors exhibit an unusual "anisotropic" geometry that hinders downstream tasks, and 2) these vectors capture more than just semantic properties, including syntactic roles and other sentence-level nuances, which can dilute the concept's true meaning. The paper aims to overcome these limitations by learning concept embeddings that specifically capture semantic properties.

Methods

The core of the paper lies in two novel contrastive learning strategies designed to fine-tune language models for better concept embeddings. The central idea is that if two sentences express similar properties about their respective concepts, their contextualized representations should be closer in the embedding space.

  1. Unsupervised Neighbourhood Strategy: This method leverages the neighborhood structure of existing contextualized word vectors. It identifies pairs of sentences that are likely to express similar properties by examining the concepts associated with the nearest neighbors of their mention vectors. If the neighbors of two mention vectors largely overlap in terms of the concepts they represent, it suggests the original mention vectors capture a similar property.
  2. ConceptNet-based Distant Supervision Strategy: This approach uses the knowledge graph ConceptNet as a source of "weak supervision." It identifies triples like "(concept, HasProperty, property)" in ConceptNet. Then, it finds sentences that mention both the concept and the property, assuming these sentences express that the concept has that property. Pairs of sentences that share a common property from ConceptNet are treated as positive examples.

Both strategies generate "weakly labelled training examples" (pairs of sentence-concept pairs that should have similar embeddings). These examples are then used to fine-tune either a linear projection of the original mention vectors (ConProj) or the BERT encoder itself (ConFT and ConCN). The fine-tuning employs a contrastive loss function that pulls similar pairs closer and pushes dissimilar pairs apart.

Datasets

The paper utilizes several standard datasets for evaluation:

  • Word Classification Benchmarks: X-McRae, CSLB, Morrow, WNSS, BabelDomains, Battig and Montague norms, and Almuhareb and Poesio dataset. These datasets contain words annotated with various semantic properties (commonsense, taxonomic, thematic).
  • Clustering Benchmark: The BM and AP datasets were used for evaluating the quality of embeddings in an unsupervised clustering setting.
  • Ontology Completion Benchmark: Five ontologies (SUMO, Wine, Economy, Olympics, and Transport) were used to evaluate the embeddings' ability to infer new rules and relationships.

The paper also leverages Wikipedia for sentence retrieval and ConceptNet for distant supervision.

Key Insights and Ideas

  • Contextualized vectors are "anisotropic": The paper highlights that standard contextualized word vectors lie in a narrow cone, which can hinder downstream tasks. Their contrastive learning approach aims to reduce this anisotropy.
  • Semantic properties are key: The core insight is that concept embeddings should focus on capturing semantic properties, not just general word meaning or syntactic roles. Contrastive learning is framed as a way to "distill" these specific properties.
  • ConceptNet as a powerful weak supervisor: The ConceptNet-based strategy (ConCN) significantly outperforms the unsupervised method, demonstrating the value of leveraging structured knowledge graphs for generating training signals.
  • Fine-tuning BERT is beneficial: While a simple projection method (ConProj) shows improvement, fine-tuning the BERT encoder itself (ConFT, ConCN) yields even better results, suggesting that adapting the underlying language model is crucial.
  • Filtering is complementary: The filtering strategy from Li et al. [31] (removing potentially idiosyncratic mention vectors) proves to be a valuable addition, offering complementary improvements to their proposed methods.

Key Results

  • State-of-the-art performance: All three proposed methods (ConProj, ConFT, ConCN) significantly outperform existing baselines on word classification, clustering, and ontology completion tasks.
  • ConCN leads the pack: The ConceptNet-based fine-tuning strategy (ConCN) consistently achieves the best results across most benchmarks.
  • ConFT > ConProj: Fine-tuning the BERT encoder (ConFT) is generally superior to learning a linear projection (ConProj).
  • CNN outperforms SVM: For word classification, a simple Convolutional Neural Network (CNN) using individual mention vectors outperforms traditional SVMs, especially on datasets where properties are rarely mentioned.
  • Reduced Anisotropy: Qualitative analysis (Figure 2) shows that the ConCN strategy leads to a reduction in the anisotropy of concept embeddings compared to the baseline "Mask" method.
  • Improved Ontology Completion: The proposed methods achieve F1 scores that are often double that of traditional skip-gram embeddings on ontology completion tasks.

Limitations and Future Work

  • Computational Cost: Fine-tuning BERT models (ConFT, ConCN) is computationally more expensive than the projection method (ConProj) or traditional methods.
  • Dependence on Language Model: The quality of the resulting embeddings is inherently tied to the capabilities of the underlying pre-trained language model.
  • Future Work: The authors suggest exploring more sophisticated ways to combine the unsupervised and distant supervision strategies, as well as investigating how to adapt these methods to other types of semantic representations beyond static concept embeddings. They also note that while their models outperform baselines, there's still room for improvement, particularly in capturing more nuanced semantic properties.

Summary updated

Abstract

Learning vectors that capture the meaning of concepts remains a fundamental challenge. Somewhat surprisingly, perhaps, pre-trained language models have thus far only enabled modest improvements to the quality of such concept embeddings. Current strategies for using language models typically represent a concept by averaging the contextualised representations of its mentions in some corpus. This is potentially sub-optimal for at least two reasons. First, contextualised word vectors have an unusual geometry, which hampers downstream tasks. Second, concept embeddings should capture the semantic properties of concepts, whereas contextualised word vectors are also affected by other factors. To address these issues, we propose two contrastive learning strategies, based on the view that whenever two sentences reveal similar properties, the corresponding contextualised vectors should also be similar. One strategy is fully unsupervised, estimating the properties which are expressed in a sentence from the neighbourhood structure of the contextualised word embeddings. The second strategy instead relies on a distant supervision signal from ConceptNet. Our experimental results show that the resulting vectors substantially outperform existing concept embeddings in predicting the semantic properties of concepts, with the ConceptNet-based strategy achieving the best results. These findings are furthermore confirmed in a clustering task and in the downstream task of ontology completion.
BibTeX
@inproceedings{li2023distilling,
  author = {Na Li and Hanane Kteich and Zied Bouraoui and Steven Schockaert},
  title = {{Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  url = {https://openreview.net/forum?id=j57WUEbt3s},
}
Figure