Representation Learning of Entities and Documents from Knowledge Base Descriptions

Ikuya Yamada, Hiroyuki Shindo, Yoshiyasu TakefujiCOLING 2018

TL;DR

This paper introduces TextEnt, a novel neural network that learns unified representations for both text documents and entities directly from knowledge bases (KBs). By training the model to predict the target entity described by a document, TextEnt effectively combines semantic signals from words and entities, achieving state-of-the-art results on fine-grained entity typing and multiclass text classification tasks.

What Problem Does This Paper Address?

The paper tackles the challenge of effectively representing both textual documents and entities in a shared, continuous vector space. Traditional methods often treat these separately or rely on complex composition functions. The authors highlight the ambiguity and variety inherent in natural language, where the same words can have different meanings and the same meaning can be expressed in multiple ways. Knowledge bases (KBs) offer a structured and unambiguous source of information, but integrating this structured knowledge with unstructured text for representation learning remains a key research area. This paper aims to bridge this gap by learning representations that capture the semantic essence of both documents and the entities they describe, leveraging the richness of KBs.

Methods

The core of the proposed method, TextEnt, is a neural network designed to learn distributed representations of entities and documents from a KB. The model takes a document, consisting of words and contextual entities, and learns to predict the target entity that the document describes. This prediction task is framed as a classification problem where the model learns to map the document and its target entity close to each other in a vector space.

Key aspects of the method include:

  • Unified Representation Learning: TextEnt learns a single vector representation for documents and entities, allowing for direct comparison and integration.
  • Predicting Target Entity: The model is trained by predicting the entity that a given document describes. This is achieved by computing a document representation (VD) that combines word-based (VDw) and contextual entity-based (VD e) representations, and then using a softmax function to predict the probability of each entity being the target.
  • Combining Word and Entity Signals: TextEnt explicitly leverages both words and contextual entities within a document to overcome ambiguity and variety in natural language.
  • Negative Sampling: To address the computational expense of summing over all KB entities in the softmax function, the authors employ negative sampling.
  • Word Dropout: To prevent overfitting, word dropout is used during training.
  • Variants: The authors also experiment with TextEnt-word (using only word representations) and TextEnt-entity (using only contextual entity representations) to demonstrate the benefit of combining both.

Datasets

The primary dataset used for training TextEnt is derived from the DBpedia NIF abstract dataset, specifically the April 2016 version. This dataset contains texts and entity annotations from the introductory sections of Wikipedia articles.

For evaluation, the paper utilizes:

  • Fine-grained Entity Typing: The dataset from Yaghoobzadeh and Schutze (2015), which consists of 201,933 Freebase entities mapped to 102 entity types. This dataset is split into training (50%), development (20%), and test (30%) sets.
  • Multiclass Text Classification:
    • 20 newsgroups dataset (20NG) (Lang, 1995): A standard dataset with 11,314 training documents and 7,532 test documents from 20 different newsgroups.
  • R8 dataset (Debole and Sebastiani, 2005): A subset of the Reuters-21578 corpus, containing 5,485 training documents and 2,189 test documents.

Key Insights and Ideas

  • Knowledge Base as a Rich Source: The paper strongly advocates for leveraging KBs not just for entity linking or knowledge graph completion, but as a rich source for learning general-purpose document and entity representations.
  • Predicting the "Subject" of a Document is Key: The core idea of training the model to predict the target entity a document describes is a clever way to force the model to learn a unified semantic space where documents and their corresponding entities are co-located. This is more direct than simply using entities as additional features.
  • Synergy of Words and Entities: The results clearly demonstrate that combining signals from both words and entities leads to superior performance compared to using either alone. This highlights the complementary nature of these information sources.
  • Generalizability Beyond KB Entities: The success on text classification tasks, which involve documents not necessarily present in the KB, suggests that the learned representations are generalizable and capture semantic meaning beyond specific KB entries.

Key Results

  • State-of-the-Art Performance: TextEnt achieved state-of-the-art results on both fine-grained entity typing and multiclass text classification tasks.
  • Fine-grained Entity Typing: TextEnt-full achieved a strong P@1 score of 93.2%, outperforming baseline models like Figment-GM and Wikipedia2Vec.
  • Multiclass Text Classification: TextEnt-full outperformed baseline models (BoW-SVM, BoE, Wikipedia2Vec, Skip-Gram-Wiki) in terms of accuracy and macro F1 score on both the 20NG and R8 datasets. For example, on 20NG, TextEnt-full achieved an accuracy of .845 and F1 of .839, compared to BoE's .831 and .827.
  • Qualitative Analysis: The model successfully encoded example sentences into vectors that were close to their relevant entities, demonstrating its ability to capture semantic relationships. For instance, a sentence about Alexei Leonov was correctly mapped to entities related to the Soviet space program.

Limitations and Future Work

  • Computational Cost: While negative sampling and other optimizations are used, training on large KBs can still be computationally intensive.
  • Dataset Size: The authors mention limiting the dataset size for computational efficiency, which might impact the full potential of the model.
  • Future Work: The authors suggest exploring the applicability of TextEnt to broader NLP tasks such as entity search and KB-based question answering. They also plan to investigate the model's performance on tasks requiring more complex composition functions.

Summary updated

BibTeX
@inproceedings{yamada2018representation,
  author = {Ikuya Yamada and Hiroyuki Shindo and Yoshiyasu Takefuji},
  title = {{Representation Learning of Entities and Documents from Knowledge Base Descriptions}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure