Deep Context: A Neural Language Model for Large-scale Networked Documents

Hao Wu, Kristina LermanIJCAI 2017 · DOI

TL;DR

This paper introduces "Deep Context Vectors" (DCV), a novel neural language model that leverages both the text within documents and the links between them to create richer document representations. By considering the "deep context" of a document, DCV significantly improves performance on tasks like document classification and link prediction, especially for short texts.

What Problem Does This Paper Address?

Traditional text mining methods often represent documents as simple word frequency vectors. While computationally efficient, this approach fails to capture the nuances of word meaning and the broader context in which words are used. More advanced methods like n-gram models and topic models incorporate local word context within a document, but they still struggle to leverage the rich semantic connections that exist between documents, especially in networked collections like Wikipedia or citation networks. The authors argue that understanding a document requires not just its internal text but also the context provided by related documents it links to, and the concepts those documents discuss. This "deep context" is crucial for accurately representing and understanding documents, particularly when documents are short and lack extensive internal text.

Methods

The core of the paper is the Deep Context Vector (DCV) model, which builds upon the log-bilinear language model (LBL). DCV extends LBL by incorporating two key sources of information:

  • Word Order within a Document: Similar to LBL, DCV considers the sequence of words in a document to learn word representations.
  • Document Network Structure: This is the novel contribution. DCV explicitly models the connections between documents. It learns a document vector for the current document and also incorporates information from neighboring documents (documents reachable within a specified number of hops, m) in the document network.

The model learns both word and document embeddings in the same low-dimensional vector space. To handle the computational complexity of considering many linked documents, they propose a Randomized Document Sequence approach, where a sequence of neighboring documents is sampled rather than exhaustively considering all neighbors. They also introduce an alternative architecture, DCV-ivLBL, which performs inverse language modeling, predicting context words from the current word and document context. For training, they employ Negative Sampling, a scalable optimization technique that avoids computing the full normalization term, making the model efficient for large datasets.

Datasets

The authors evaluate their model on three large-scale, diverse datasets:

  • Wikipedia: A large collection of interlinked web pages, providing a dense document network with rich textual content.
  • DBLP: A dataset of academic papers, characterized by very short documents (titles) and a sparser citation network. This dataset is particularly challenging for text-only models.
  • Legal: Digitized federal court opinions, featuring long documents and a sparse citation network.

Key Insights and Ideas

  • Links are Crucial for Short Texts: The most striking insight is how effectively DCV leverages document links to compensate for the lack of textual information in short documents (like DBLP paper titles). This demonstrates that inter-document relationships can provide significant semantic context.
  • "Deep Context" Matters: The paper validates the hypothesis that a document's meaning is enriched by its connections to other documents. This "deep context" allows for more robust representations than text alone.
  • Scalability is Achieved: Despite incorporating complex relational information, the DCV model, particularly with negative sampling and randomized document sequences, proves to be scalable to millions of documents and billions of links.
  • Low-Dimensional Embeddings are Powerful: DCV achieves competitive or superior performance to methods using millions of features (like BOW) with vectors of only a few hundred dimensions, highlighting the efficiency and effectiveness of learned representations.

Key Results

  • Document Classification: DCV consistently outperforms baseline methods (BOW, LDA, Skip-gram, PV, DeepWalk, LINE) across all three datasets, especially on DBLP where it shows a significant performance gain. For instance, on DBLP, DCV's performance is notably better than PV, which struggles with short texts.
  • Link Prediction: DCV also demonstrates strong performance in predicting unseen links between documents, outperforming other methods, particularly on the Wikipedia dataset.
  • Dimensionality: Classification performance generally improves with vector dimensionality up to 200-400 dimensions, with diminishing returns beyond that.
  • Number of Hops: For link prediction, a small number of hops (m=1 or 2) is often sufficient and more computationally efficient, with performance not significantly degrading for larger hop counts in some cases.
  • Training Time: DCV is shown to be scalable, with training times on the Legal dataset (2M documents, 6.3B words, 4M links) ranging from 10 to 26 hours, which is considered efficient.

Limitations and Future Work

  • Computational Cost of Hops: While randomized sequences help, considering a large number of hops (m) can still be computationally expensive and may not always yield proportional performance gains.
  • Tuning for Long Documents: The authors note that DCV can be "not easily tuned" for very long documents (like Legal opinions), though it still performs well.
  • Future Work: The authors suggest exploring "side information" such as the relative position of words, hyperlink structure within paragraphs, and more sophisticated ways to determine which linked documents are most relevant for modeling each word. They also mention that higher-order random walks could be explored for sampling document neighborhoods.

Summary updated

BibTeX
@inproceedings{wu2017deep,
  author = {Hao Wu and Kristina Lerman},
  title = {{Deep Context: A Neural Language Model for Large-scale Networked Documents}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/431},
}
Figure