Gaussian Embedding of Linked Documents from a Pretrained Semantic Space

Antoine Gourru, Julien Velcin, Julien JacquesIJCAI 2020 · DOI

TL;DR

This paper introduces GELD, a novel method for representing documents as Gaussian distributions in a semantic space, capturing both their textual content and their relationships within a network. By modeling uncertainty, GELD achieves state-of-the-art performance in document classification and link prediction tasks, offering a more nuanced understanding of document representations.

What Problem Does This Paper Address?

Documents are increasingly interconnected, forming networks like citation graphs or social media connections. Simply representing documents as points in a semantic space (like word embeddings) overlooks the rich information embedded in these links and the inherent uncertainty in document meaning. Existing methods either fail to jointly model text and links, don't represent words and documents in the same space, or don't adequately capture document uncertainty. This paper aims to bridge this gap by proposing a method that embeds documents as Gaussian distributions, allowing for a more robust and informative representation that accounts for both textual variance and network relationships.

Methods

The core of GELD is a generative model that treats each document as a Gaussian distribution in a pretrained semantic space. This means each document is characterized by a mean vector (its primary representation) and a diagonal variance vector, which quantifies the uncertainty associated with its meaning. This uncertainty is influenced by two factors: the diversity of words used within the document and the diversity of documents it cites. The model is trained using a maximum likelihood estimation approach, optimizing a weighted log-likelihood function that balances the contribution of textual information and network structure. The authors employ a Robbins-Monro method for optimization to prevent premature convergence to suboptimal solutions.

Datasets

The paper evaluates GELD on three datasets:

  • Cora: A citation network of scientific documents.
  • Dblp: Another citation network derived from computer science publications.
  • Nyt: A corpus of news articles.

These datasets are preprocessed by removing stop words and filtering rare words to create a vocabulary. The authors do not introduce new benchmarks but rather use these established datasets to compare GELD against existing methods.

Key Insights and Ideas

  • Modeling Uncertainty is Crucial: Representing documents as Gaussian distributions, rather than just points, allows GELD to capture the inherent ambiguity and variability in document meaning, leading to improved performance.
  • Joint Representation of Words and Documents: GELD embeds both words and documents in the same semantic space, enabling direct comparison and facilitating tasks like query enhancement or cluster description.
  • Variance Reflects Semantic Richness: The variance of a document's Gaussian representation is shown to correlate with its semantic breadth. Documents citing diverse sources or using a wider range of vocabulary tend to have higher variance.
  • Impact of Variance on Learning: The GELD model implicitly down-weights documents with high variance during parameter updates, suggesting that uncertain or overly general documents have less influence on the learning process.

Key Results

GELD consistently outperforms or matches state-of-the-art methods across various tasks and datasets:

  • Classification: GELD achieves superior Micro-F1 scores on Cora and Dblp, outperforming methods like TADW by up to 7 points on Dblp. It also performs competitively on the Nyt dataset.
  • Link Prediction: GELD demonstrates strong performance in link prediction, outperforming most baselines on Cora and Dblp. While comparable to TADW on Nyt, it shows more consistent results across different network topologies.
  • Qualitative Analysis: The paper provides compelling qualitative insights, showing that words closest to class centroids in Cora are semantically relevant to those classes. Additionally, documents with higher variance are shown to be those that apply concepts (like Latent Dirichlet Allocation) to broader domains.

Limitations and Future Work

The authors acknowledge that while GELD represents documents as Gaussian distributions, the variance is learned through a maximum likelihood estimation. They suggest future work could involve developing a fully Bayesian version of the model to incorporate priors on the mean and variance. Additionally, they propose exploring how to directly integrate the learned variance into downstream tasks, rather than solely relying on the document mean as is done in the current evaluation.

Summary updated

BibTeX
@inproceedings{gourru2020gaussian,
  author = {Antoine Gourru and Julien Velcin and Julien Jacques},
  title = {{Gaussian Embedding of Linked Documents from a Pretrained Semantic Space}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/541},
}
Figure