Modelling Semantic Categories Using Conceptual Neighborhood

Zied Bouraoui, José Camacho-Collados, Luis Espinosa Anke, Steven Schockaert

Abstract

While many methods for learning vector space embeddings have been proposed in the field of Natural Language Processing, these methods typically do not distinguish between categories and individuals. Intuitively, if individuals are represented as vectors, we can think of categories as (soft) regions in the embedding space. Unfortunately, meaningful regions can be difficult to estimate, especially since we often have few examples of individuals that belong to a given category. To address this issue, we rely on the fact that different categories are often highly interdependent. In particular, categories often have conceptual neighbors, which are disjoint from but closely related to the given category (e.g. fruit and vegetable). Our hypothesis is that more accurate category representations can be learned by relying on the assumption that the regions representing such conceptual neighbors should be adjacent in the embedding space. We propose a simple method for identifying conceptual neighbors and then show that incorporating these conceptual neighbors indeed leads to more accurate region based representations.

Introduction

In Natural Language Processing (NLP), entities are typically represented as points in a high-dimensional vector space. While this works well for individuals (e.g., "carrot"), it is less intuitive for categories (e.g., "vegetable"). A category is more naturally modeled as a region encompassing many individuals.

The challenge lies in the "small data" problem: defining these regions accurately requires significant data. If a model only sees a few examples of a category, it will likely underestimate its true scope. Humans, however, learn categories not just by what they are, but by what they are not, often comparing them to similar, disjoint categories. This paper introduces the concept of "conceptual neighborhood"—the idea that related categories share boundaries and compete for space—to improve category induction from limited datasets.

The SECOND Framework

The authors propose SECOND (SEmantic categories with COnceptual NeighborhooD), a three-step framework to model categories as regions:

  1. Predicting Neighbors from Embeddings: By analyzing categories with many known instances, the model identifies which categories are likely adjacent in vector space.
  2. Predicting Neighbors from Text: A classifier identifies conceptual neighbors based on co-occurrence patterns in text (e.g., Wikipedia). Interestingly, simple word embedding averaging (WEA) was found to be more robust than contextualized BERT embeddings for this task.
  3. Category Induction: To model a target category $C$, the system identifies its $k$ most likely conceptual neighbors. It then uses a Gaussian model to capture the local domain and a multinomial logistic regression to "carve up" that domain into specific regions.
Conceptual neighborhood: Category C is defined by its proximity to neighbors N1-N4.

Key Insights and Results

A central hypothesis of this work is the "Betweenness" Assumption: if two categories are conceptual neighbors, the space between their known examples likely belongs to one of them rather than being empty. By defining boundaries relative to neighbors, the model can correctly classify "outliers"—instances far from their category's center but clearly distinct from neighbors.

Experimental evaluation on a category induction task using BabelNet and Wikipedia data demonstrated that incorporating conceptual neighbors leads to a massive improvement over standard isolated models.

Performance comparison of SECOND variants against standard Gaussian and Multinomial baselines.

As shown in the results, the SECOND model significantly outperformed baselines. For instance, using four neighbors (SECOND-WEA4), the model achieved an F1 score of 54.2%, compared to just 22.3% for a standard Gaussian model. Even a single neighbor (k=1) provided a substantial performance boost (~47% F1), confirming the strength of the neighborhood signal.

Conclusion

By leveraging the interdependence of categories, the SECOND framework provides a robust method for learning category regions from sparse data. While currently dependent on existing taxonomies and focused on disjoint neighbors, it opens new avenues for modeling complex lexical relations and vague semantic boundaries in vector spaces.

Summary updated

Abstract

While many methods for learning vector space embeddings have been proposed in the field of Natural Language Processing, these methods typically do not distinguish between categories and individuals. Intuitively, if individuals are represented as vectors, we can think of categories as (soft) regions in the embedding space. Unfortunately, meaningful regions can be difficult to estimate, especially since we often have few examples of individuals that belong to a given category. To address this issue, we rely on the fact that different categories are often highly interdependent. In particular, categories often have conceptual neighbors, which are disjoint from but closely related to the given category (e.g. fruit and vegetable). Our hypothesis is that more accurate category representations can be learned by relying on the assumption that the regions representing such conceptual neighbors should be adjacent in the embedding space. We propose a simple method for identifying conceptual neighbors and then show that incorporating these conceptual neighbors indeed leads to more accurate region based representations.
BibTeX
@inproceedings{bouraoui2020modelling,
  author = {Zied Bouraoui and José Camacho-Collados and Luis Espinosa Anke and Steven Schockaert},
  title = {{Modelling Semantic Categories Using Conceptual Neighborhood}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=SXxTtf18Ns},
}
Figure