Utilizing Context in Generative Bayesian Models for Linked Corpus

Saurabh Kataria, Prasenjit Mitra, Sumit BhatiaAAAI 2010 · OpenReview

Abstract

In an interlinked corpus of documents, the context in which a citation appears provides extra information about the cited document. However, associating terms in the context to the cited document remains an open problem. We propose a novel document generation approach that statistically incorporates the context in which a document links to another document. We quantitatively show that the proposed generation scheme explains the linking phenomenon better than previous approaches. The context information along with the actual content of the document provides significant improvements over the previous approaches for various real world evaluation tasks such as link prediction and log-likelihood estimation on unseen content. The proposed method is more scalable to large collection of documents compared to the previous approaches.

1. Introduction

In large, interlinked document collections—such as the World Wide Web or academic databases—documents contain both text and structural links (citations). Traditional topic models like Latent Dirichlet Allocation (LDA) focus exclusively on text, ignoring the rich structural information provided by links. While recent models like linked-LDA and link-PLSA-LDA attempt to model text and links jointly, they treat links as global properties of the document. Specifically, they identify what is being cited but ignore how it is being cited.

When an author cites a paper or links to a webpage, the specific words immediately surrounding that link—the "citation context"—are highly indicative of the linked document's topic. By ignoring this local context, existing models fail to capture valuable semantic information. This paper introduces a mathematical framework to model the relationship between a citation and its surrounding text to better uncover latent topics.

2. Methodology

The authors propose two new generative Bayesian models: cite-LDA and cite-PLSA-LDA. These models adapt previous link-aware architectures to explicitly handle citation context.

Graphical representation of (a) linked-LDA, (b) link-PLSA-LDA, (c) cite-LDA, and (d) cite-PLSA-LDA.

The generative process defines a "citation context" as a window of words appearing immediately before and after a link. The model operates on two primary assumptions:

  1. Words falling outside a citation context have topics drawn from the document's overall topic mixture.
  2. Words falling inside a citation context share a specific, narrow topic with the citation link itself.

Because exact inference is intractable, Gibbs sampling is used to approximate the posterior distribution. A significant advantage of this formulation is that sampling updates tie the citation variable explicitly to word variables. Consequently, the algorithm's time complexity grows linearly with the number of words in the corpus rather than the number of links, enabling scalability for web-scale datasets.

3. Experimental Evaluation

The models were evaluated on two benchmark datasets:

  • CiteSeer: 3,312 scientific documents across 6 research fields with 4,132 citation links. A context window of 60 words (radius of 30) was extracted around each citation.
  • WebKB: 2,877 computer science department web pages with 1,764 hyperlinks. Context was defined by anchor text and surrounding words.

4. Results and Discussion

The proposed models were tested on log-likelihood estimation and link prediction.

In the link prediction task, models were required to rank potential outgoing citations for unseen documents. As shown in Figure 6, cite-PLSA-LDA consistently achieved the lowest (best) rank for true cited documents across varying numbers of topics.

Comparison of link prediction performance (Rank at 100% Recall) across different models.

4.2 Generalization and Domain Insights

On unseen text, cite-PLSA-LDA achieved the highest predictive log-likelihood, outperforming standard LDA and previous link-aware models. Interestingly, the performance gain was more pronounced on CiteSeer than on WebKB. This suggests that academic citations are more context-rich and specific compared to the often noisy nature of web links.

The authors also observed that the "effective" context radius varies by domain. In academic papers, topic correlation remains strong up to 10–14 words from the citation, whereas web page context relevance drops off after approximately 6 words.

5. Conclusion

By treating citation context as a semantic bottleneck, the proposed models align the latent spaces of words and links more effectively than document-wide distributions. Furthermore, the word-tying approach provides a scalable solution for large-scale linked corpora. Future work will focus on mechanisms for the automatic selection of the optimal context radius to handle diverse document types without manual hyperparameter tuning.

Summary updated

Abstract

In an interlinked corpus of documents, the context in which a citation appears provides extra information about the cited document. However, associating terms in the context to the cited document remains an open problem. We propose a novel document generation approach that statistically incorporates the context in which a document links to another document. We quantitatively show that the proposed generation scheme explains the linking phenomenon better than previous approaches. The context information along with the actual content of the document provides signicant improvements over the previous approaches for various real world evaluation tasks such as link prediction and log-likelihood estimation on unseen content. The proposed method is more scalable to large collection of documents compared to the previous approaches.
BibTeX
@inproceedings{kataria2010utilizing,
  author = {Saurabh Kataria and Prasenjit Mitra and Sumit Bhatia},
  title = {{Utilizing Context in Generative Bayesian Models for Linked Corpus}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2010},
  url = {https://openreview.net/forum?id=HkZ-HkWuWB},
}
Figure