A Mixture-of-Experts Model for Learning Multi-Facet Entity Embeddings

Rana Alshaikh, Zied Bouraoui, Shelan S. Jeawak, Steven SchockaertCOLING 2020 · OpenReview

TL;DR

This paper introduces a novel "Mixture-of-Experts" (MoE) model for learning entity embeddings that explicitly capture multiple facets of similarity. By training specialized "experts" for different aspects of an entity's description, their MoE approach significantly outperforms standard single-vector embeddings on various downstream tasks, demonstrating the benefit of disentangling complex similarity notions.

What Problem Does This Paper Address?

Entity embeddings, widely used in NLP for tasks like recommendation and knowledge injection, typically represent each entity as a single vector. However, similarity between entities is often multi-faceted (e.g., two movies can be similar by genre or by actors). Standard embeddings struggle to capture these distinct facets, leaving it to downstream applications to disentangle them. This can be suboptimal, especially for tasks like concept induction where identifying the right facet of similarity is crucial. Existing methods for learning facet-specific embeddings often rely on computationally expensive techniques like Multi-Dimensional Scaling (MDS) or make strong assumptions about how facets manifest in word embeddings.

Methods

The core of this paper is a Mixture-of-Experts (MoE) framework adapted for learning entity embeddings. Instead of a single embedding vector per entity, the model learns multiple, specialized embedding vectors (experts). Each expert is designed to capture a different "facet" or aspect of the entity's description.

The model leverages a GloVe (Global Vectors for Word Representation) variant for learning these expert embeddings. A key innovation is the gating network, which uses pre-trained word vectors to dynamically assign words to specific experts. This gating mechanism is crucial: it encourages words relevant to the same facet to have similar word embeddings, thereby promoting semantic coherence within each expert's learned facet. This approach avoids the strong assumption that all words for a facet must form a single cluster, as in some prior work.

The training process uses an Expectation-Maximization (EM) algorithm. The E-step estimates the probability of a word belonging to each expert, while the M-step updates the parameters of both the gating network and the experts.

Datasets

The authors evaluate their model on several existing datasets and one newly compiled dataset:

  • Movies: Contains bag-of-words (BoW) descriptions from movie reviews.
  • Place types: BoW descriptions from Flickr tags.
  • Buildings: BoW descriptions from Wikipedia articles.
  • Locations: A newly compiled dataset of geographic locations in the UK, using Flickr tags tokenized by Wordninja and filtered for pre-trained word vectors. This dataset is larger than the others.
  • Wikipedia: A large dataset of Wikipedia concepts, with tasks involving predicting Wikidata semantic types and nine Wikidata attributes.

Key Insights and Ideas

  • Similarity is inherently multi-faceted: The paper strongly argues that a single vector representation is insufficient for capturing the complex ways entities can be similar.
  • MoE for disentanglement: The MoE framework provides a natural way to learn specialized representations for different aspects of an entity.
  • Gating network with word vectors: Using pre-trained word vectors in the gating network is a clever way to inject inductive bias, guiding the model to learn semantically coherent facets without requiring explicit supervision for facet assignment.
  • Scalability: The proposed method is designed to be more scalable than previous MDS-based approaches, making it applicable to larger datasets.
  • Interpretability: While not the primary focus, the facet-specific embeddings offer potential for better interpretability, as demonstrated by the qualitative analysis of nearest neighbors and word assignments to experts.

Key Results

The paper demonstrates the effectiveness of their MoE model across various classification tasks on the evaluated datasets.

  • Quantitative Performance: The MoEGloVe model consistently outperforms the standard GloVe baseline and other facet-based methods (IncAggGloVe, IncHDBGloVe) in terms of F1 score across multiple classification tasks. For instance, on the Place types dataset, MoEGloVe achieves F1 scores of 0.45, 0.48, and 0.45 for the Fours., Geo., and OpenC. attributes, respectively, which are notably higher than the GloVe baseline (0.34, 0.23, 0.26). Similar improvements are observed on the Movies and Buildings datasets.
  • Wikipedia and Locations: The MoEGloVe model also shows significant improvements on the larger Wikipedia and Locations datasets, further highlighting its scalability and effectiveness.
  • Qualitative Analysis: Table 4 provides examples of nearest neighbors for movies. When using a genre-specific facet, the neighbors are more consistently related by genre compared to the full space, where other similarities (e.g., actors, directors) can also influence the results. Figure 1 visually demonstrates how different facets can capture distinct spatial similarities for locations.

Limitations and Future Work

  • Focus on BoW: The current model is primarily designed for Bag-of-Words (BoW) representations. The authors suggest that in future work, similar strategies could be applied to document embedding methods based on models like BERT or other language models.
  • Gating Network Input: While the current gating network uses word vectors, the authors mention that incorporating entity information into the gating network could potentially improve facet identification, though preliminary experiments did not yield better results.
  • Interpretability of Dimensions: The paper focuses on learning facet-specific spaces rather than ensuring individual dimensions within those spaces are interpretable. While the facets themselves are shown to be meaningful, further work could explore methods to achieve more granular interpretability of the learned dimensions.

Summary updated

Abstract

Various methods have already been proposed for learning entity embeddings from text descriptions. Such embeddings are commonly used for inferring properties of entities, for recommendation and entity-oriented search, and for injecting background knowledge into neural architectures, among others. Entity embeddings essentially serve as a compact encoding of a similarity relation, but similarity is an inherently multi-faceted notion. By representing entities as single vectors, existing methods leave it to downstream applications to identify these different facets, and to select the most relevant ones. In this paper, we propose a model that instead learns several vectors for each entity, each of which intuitively captures a different aspect of the considered domain. We use a mixture-of-experts formulation to jointly learn these facet-specific embeddings. The individual entity embeddings are learned using a variant of the GloVe model, which has the advantage that we can easily identify which properties are modelled well in which of the learned embeddings. This is exploited by an associated gating network, which uses pre-trained word vectors to encourage the properties that are modelled by a given embedding to be semantically coherent, i.e. to encourage each of the individual embeddings to capture a meaningful facet.
BibTeX
@inproceedings{alshaikh2020mixture,
  author = {Rana Alshaikh and Zied Bouraoui and Shelan S. Jeawak and Steven Schockaert},
  title = {{A Mixture-of-Experts Model for Learning Multi-Facet Entity Embeddings}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=vggwr6FOag},
}
Figure