Representation Learning for Measuring Entity Relatedness with Rich Information

Yu Zhao, Zhiyuan Liu, Maosong Sun

Introduction

Measuring semantic relatedness between entities—such as the conceptual link between stock and petroleum—is a cornerstone of information retrieval and natural language processing. While Wikipedia provides a wealth of data for this task, most existing methodologies are "homogeneous," focusing on a single signal like hyperlink structure or textual content. This approach often suffers from data sparsity and ignores the complex interactions between different data types. This paper presents Coordinate Matrix Factorization (CMF), a framework designed to integrate multiple relational signals into a unified latent semantic space.

Methodology

The CMF Framework

The CMF model treats entity relatedness as a matrix completion problem. It maps entities, categories, and words into a shared $K$-dimensional vector space. The framework constructs three primary association matrices to capture the observed relationships within Wikipedia:

  • X (Entity-Entity): Derived from hyperlink structures.
  • Y (Entity-Category): Derived from taxonomic labels.
  • Z (Entity-Word): Derived from textual content.
Overview of the CMF framework mapping semantic relationships from the semantic space to a low-rank vector space.

Overcoming Sparsity

To address the issue of sparsity—where related entities may not be directly linked—the authors employ Random Walks to simulate paths through the Wikipedia graph. This captures implicit co-occurrences between semantically related but disconnected vertices. These sequences are transformed using Positive Pointwise Mutual Information (PPMI) to populate the association matrices before factorization. The model is optimized using a coordinate stochastic gradient descent scheme to minimize a joint non-zero square loss function.

Experimental Setup and Results

Dataset

The model was trained on the Chinese Wikipedia corpus, encompassing approximately 291,000 entities, 156,000 categories, and 365,000 unique words. Evaluation was conducted using the Words-240 dataset, consisting of entity pairs with human-assigned similarity ratings.

Performance Metrics

Performance was measured using Spearman’s $\rho$ correlation against human judgments. As shown in the results table, the full CMF model (utilizing X, Y, and Z matrices) achieved a correlation of 0.465, significantly outperforming traditional graph-based measures and contemporary representation learning baselines like DeepWalk and Skip-gram.

Comparison of Spearman’s correlation coefficients (ρ × 100) for CMF variants and baseline models.

Discussion and Key Insights

  • Multi-relational Synergy: The integration of all three relation types (links, categories, and words) consistently yielded the best performance, validating the hypothesis that heterogeneous data sources provide complementary information.
  • The Power of Taxonomy: Interestingly, entity-category relations ($Y$) proved more beneficial for measuring relatedness than entity-word relations ($Z$), despite the word matrix containing significantly more non-zero entries. This suggests that categorical labels provide a more concentrated semantic signal than raw text.
  • Limitations in Implicit Knowledge: While CMF excels at capturing relations explicitly documented in an encyclopedia (e.g., chemistry and atom), it struggles with experiential or "common sense" relations that humans find intuitive but are rarely co-located in formal text (e.g., movie and popcorn).

Conclusion

Coordinate Matrix Factorization provides a robust framework for learning entity representations by leveraging the rich, multi-faceted structure of Wikipedia. By bridging the gap between textual content and structural links, CMF achieves state-of-the-art results in semantic relatedness tasks. Future research will focus on incorporating external common-sense knowledge bases to better capture implicit human associations.

Summary updated

Abstract

Incorporating multiple types of relational information from heterogeneous networks has been proved effective in data mining. Although Wikipedia is one of the most famous heterogeneous network, previous works of semantic analysis on Wikipedia are mostly limited on single type of relations. In this paper, we aim at incorporating multiple types of relations to measure the semantic relatedness between Wikipedia entities. We propose a framework of coordinate matrix factorization to construct lowdimensional continuous representation for entities, categories and words in the same semantic space. We formulate this task as the completion of a sparse entity-entity association matrix, in which each entry quantifies the strength of relatedness between corresponding entities. We evaluate our model on the task of judging pair-wise word similarity. Experiment result shows that our model outperforms both traditional entity relatedness algorithms and other representation learning models.
BibTeX
@inproceedings{zhao2015representation,
  author = {Yu Zhao and Zhiyuan Liu and Maosong Sun},
  title = {{Representation Learning for Measuring Entity Relatedness with Rich Information}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure