Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings

Hanane Kteich, Na Li, Usashi Chatterjee, Zied Bouraoui, Steven Schockaert

Abstract

Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e. sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g. the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.

Introduction

Traditional concept embeddings, typically derived from language models like BERT or GloVe, excel at capturing taxonomic relationships—for instance, recognizing that a "banana" is a "fruit." However, they frequently struggle to capture other "facets" of meaning. A standard embedding might fail to identify that a "banana" and a "school bus" share the property of being "yellow," or that a "refrigerator" and "Antarctica" are related by the facet of "temperature."

This limitation hinders knowledge engineering tasks such as ontology expansion and ultra-fine entity typing, which require identifying concepts sharing specific, often non-taxonomic, properties. While Large Language Models (LLMs) possess this knowledge, their direct application for large-scale extraction is often computationally expensive or unreliable. This research introduces efficient embeddings designed to understand these diverse facets of commonality.

Methodology: Facets as Masks

The authors propose a bi-encoder architecture based on BERT, utilizing a novel "facets as masks" mechanism. Instead of learning multiple separate vectors for a single concept, the model learns a single high-dimensional concept embedding and employs three encoders:

  • Con(c): Maps a concept to its base embedding.
  • Prop(p): Maps a property (e.g., "yellow") to an embedding.
  • Facet(p): Maps a property to a mask—a vector of the same dimensionality as the concept embedding.

To determine if a concept possesses a specific property, a component-wise product is performed between the concept embedding and the facet mask. This operation highlights the coordinates in the embedding space relevant to that specific facet, such as color or shape. Training involves a dual-loss approach: a standard contrastive loss for property prediction and an InfoNCE loss to ensure properties within the same facet (e.g., "red" and "blue" under "color") produce similar masks. Supervision is facilitated by LLMs, which generate the necessary (property, facet) pairs.

Experimental Results

The proposed multi-facet model (BiEnc-F) was evaluated across several benchmarks, including property prediction, ontology completion, and ultra-fine entity typing.

Property Prediction and Outlier Detection

In property prediction tasks using the McRae and CSLB datasets, BiEnc-F consistently outperformed standard bi-encoders across all evaluation splits. The use of BERT-large (BL) as a backbone provided the most significant performance gains.

Property prediction results on McRae and CSLB datasets.

The model's ability to identify non-taxonomic commonalities was further tested on a new outlier detection benchmark. The multi-facet approach, denoted as MultiConEmb in this context, showed dramatic improvements over baselines. For example, in identifying "edible" items, the success rate increased from 23% to 67%, and for "used for cooking," it jumped from 59% to 93%.

Outlier detection performance across various non-taxonomic properties.

Downstream Task Performance

On the Ultra-Fine Entity Typing (UFET) task, the multi-facet clustering strategy (MClu) achieved a state-of-the-art F1 score of 51.3. This demonstrates that structuring the label space with facets significantly aids generalization to rare labels.

Conclusion and Future Work

This work demonstrates that explicitly modeling facets through a masking mechanism allows concept embeddings to capture a broader spectrum of commonsense knowledge. While the current implementation focuses on general concepts and uses BERT-based encoders, future research could explore applying this multi-facet approach to named entities or document retrieval. Additionally, investigating hierarchical facet structures remains a promising avenue for further enhancing the granularity of these representations.

Summary updated

Abstract

Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e. sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g. the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.
BibTeX
@inproceedings{kteich2024modelling,
  author = {Hanane Kteich and Na Li and Usashi Chatterjee and Zied Bouraoui and Steven Schockaert},
  title = {{Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=j2ci8t7kay},
}

Loading related content...

Figure