Abstract
Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e. sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g. the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.
Introduction
Traditional concept embeddings, typically derived from language models like BERT or GloVe, excel at capturing taxonomic relationships—for instance, recognizing that a "banana" is a "fruit." However, they frequently struggle to capture other "facets" of meaning. A standard embedding might fail to identify that a "banana" and a "school bus" share the property of being "yellow," or that a "refrigerator" and "Antarctica" are related by the facet of "temperature."
This limitation hinders knowledge engineering tasks such as ontology expansion and ultra-fine entity typing, which require identifying concepts sharing specific, often non-taxonomic, properties. While Large Language Models (LLMs) possess this knowledge, their direct application for large-scale extraction is often computationally expensive or unreliable. This research introduces efficient embeddings designed to understand these diverse facets of commonality.
Methodology: Facets as Masks
The authors propose a bi-encoder architecture based on BERT, utilizing a novel "facets as masks" mechanism. Instead of learning multiple separate vectors for a single concept, the model learns a single high-dimensional concept embedding and employs three encoders:
- Con(c): Maps a concept to its base embedding.
- Prop(p): Maps a property (e.g., "yellow") to an embedding.
- Facet(p): Maps a property to a mask—a vector of the same dimensionality as the concept embedding.
To determine if a concept possesses a specific property, a component-wise product is performed between the concept embedding and the facet mask. This operation highlights the coordinates in the embedding space relevant to that specific facet, such as color or shape. Training involves a dual-loss approach: a standard contrastive loss for property prediction and an InfoNCE loss to ensure properties within the same facet (e.g., "red" and "blue" under "color") produce similar masks. Supervision is facilitated by LLMs, which generate the necessary (property, facet) pairs.
Experimental Results
The proposed multi-facet model (BiEnc-F) was evaluated across several benchmarks, including property prediction, ontology completion, and ultra-fine entity typing.
Property Prediction and Outlier Detection
In property prediction tasks using the McRae and CSLB datasets, BiEnc-F consistently outperformed standard bi-encoders across all evaluation splits. The use of BERT-large (BL) as a backbone provided the most significant performance gains.
The model's ability to identify non-taxonomic commonalities was further tested on a new outlier detection benchmark. The multi-facet approach, denoted as MultiConEmb in this context, showed dramatic improvements over baselines. For example, in identifying "edible" items, the success rate increased from 23% to 67%, and for "used for cooking," it jumped from 59% to 93%.
Downstream Task Performance
On the Ultra-Fine Entity Typing (UFET) task, the multi-facet clustering strategy (MClu) achieved a state-of-the-art F1 score of 51.3. This demonstrates that structuring the label space with facets significantly aids generalization to rare labels.
Conclusion and Future Work
This work demonstrates that explicitly modeling facets through a masking mechanism allows concept embeddings to capture a broader spectrum of commonsense knowledge. While the current implementation focuses on general concepts and uses BERT-based encoders, future research could explore applying this multi-facet approach to named entities or document retrieval. Additionally, investigating hierarchical facet structures remains a promising avenue for further enhancing the granularity of these representations.
Summary updated