Dynamic Embedding on Textual Networks via a Gaussian Process

Pengyu Cheng, Yitong Li, Xinyuan Zhang, Liqun Chen, David E. Carlson, Lawrence Carin

Abstract

Textual network embedding aims to learn low-dimensional representations of text-annotated nodes in a graph. Prior work in this area has typically focused on fixed graph structures; however, real-world networks are often dynamic. We address this challenge with a novel end-to-end node-embedding model, called Dynamic Embedding for Textual Networks with a Gaussian Process (DetGP). After training, DetGP can be applied efficiently to dynamic graphs without re-training or backpropagation. The learned representation of each node is a combination of textual and structural embeddings. Because the structure is allowed to be dynamic, our method uses the Gaussian process to take advantage of its non-parametric properties. To use both local and global graph structures, diffusion is used to model multiple hops between neighbors. The relative importance of global versus local structure for the embeddings is learned automatically. With the non-parametric nature of the Gaussian process, updating the embeddings for a changed graph structure requires only a forward pass through the learned model. Considering link prediction and node classification, experiments demonstrate the empirical effectiveness of our method compared to baseline approaches. We further show that DetGP can be straightforwardly and efficiently applied to dynamic textual networks.

1. Introduction

Most graph embedding techniques are designed for static networks. However, real-world networks—such as social media platforms, citation databases, and communication networks—are inherently dynamic. New users join, new papers are published, and relationships evolve constantly.

Existing "textual network" embedding methods, which utilize both the graph topology and the text associated with nodes, typically require the entire model to be re-trained from scratch whenever the graph structure changes. This is computationally prohibitive for large-scale, real-time applications. While inductive methods like GraphSAGE exist, they often struggle to integrate rich textual information effectively or fail to capture global graph structures. DetGP (Dynamic Embedding for Textual Networks with a Gaussian Process) addresses these gaps by leveraging the non-parametric nature of Gaussian Processes to generate embeddings for new nodes or edges via a single, efficient forward pass.

2. Methodology

DetGP treats the relationship between text and graph structure through the lens of a Gaussian Process (GP), shifting the focus from learning a fixed table of embeddings to learning a functional mapping from text to graph positions.

2.1 Joint Text-Structure Encoding

The model architecture comprises a text encoder (e.g., word embedding averaging) that transforms raw text into a "textual embedding." A GP then maps these textual features into a "structural embedding" space. The final network embedding is a combination of these two components.

The DetGP architecture jointly models textual and structural information through a text encoder and a structure embedding layer.

2.2 Multi-hop Diffusion and Scalability

To capture both local (immediate neighbors) and global (distant neighbors) information, DetGP utilizes a learnable multi-hop transition matrix. This allows the model to automatically weight how much "context" from the rest of the graph should influence a specific node's representation.

To overcome the $O(N^3)$ computational complexity inherent in standard GPs, DetGP employs "inducing points." These are a small set of learnable landmark points that act as proxies for the entire dataset, significantly reducing complexity and allowing the model to scale to larger graphs.

3. Experimental Evaluation

The model was evaluated on three standard citation network benchmarks: Cora (2,277 nodes), DBLP (60,744 nodes), and HepTh (1,038 nodes).

DetGP demonstrated superior performance in link prediction tasks, particularly when training data was sparse. On the Cora dataset with only 15% of edges known, DetGP achieved an AUC of 93.4, significantly outperforming structural-only methods like DeepWalk and node2vec, which achieved scores in the mid-50s.

Link prediction performance (AUC) on Cora and HepTh datasets across varying training edge percentages, showing DetGP's consistent lead.

3.2 Node Classification and Dynamic Efficiency

On the DBLP dataset, DetGP achieved a Macro-F1 score of 79.8, outperforming all baselines including WANE and CANE. Furthermore, in "cold-start" scenarios—predicting for nodes never seen during training—DetGP showed remarkable robustness. On Cora with only 10% training nodes, DetGP achieved a link prediction AUC of 62.9, surpassing GraphSAGE’s 62.1.

4. Discussion and Insights

A key insight from the research is the role of inducing points as prototypes. t-SNE visualizations reveal that these points naturally distribute themselves to cover different categories or clusters within the network, acting as anchors for the embedding space. Additionally, the model's ability to learn structural weighting ($\alpha$) for different hops proved vital; in Cora experiments, local and second-order transitions were found to be the most informative for classification.

While DetGP is highly efficient, its performance remains sensitive to the quality of the underlying text encoder. Future work may explore integrating more complex encoders like BERT or extending the framework to other attributed graphs involving images or numerical sensor data.

Summary updated

Abstract

Textual network embedding aims to learn low-dimensional representations of text-annotated nodes in a graph. Prior work in this area has typically focused on fixed graph structures; however, real-world networks are often dynamic. We address this challenge with a novel end-to-end node-embedding model, called Dynamic Embedding for Textual Networks with a Gaussian Process (DetGP). After training, DetGP can be applied efficiently to dynamic graphs without re-training or backpropagation. The learned representation of each node is a combination of textual and structural embeddings. Because the structure is allowed to be dynamic, our method uses the Gaussian process to take advantage of its non-parametric properties. To use both local and global graph structures, diffusion is used to model multiple hops between neighbors. The relative importance of global versus local structure for the embeddings is learned automatically. With the non-parametric nature of the Gaussian process, updating the embeddings for a changed graph structure requires only a forward pass through the learned model. Considering link prediction and node classification, experiments demonstrate the empirical effectiveness of our method compared to baseline approaches. We further show that DetGP can be straightforwardly and efficiently applied to dynamic textual networks.
BibTeX
@inproceedings{cheng2020dynamic,
  author = {Pengyu Cheng and Yitong Li and Xinyuan Zhang and Liqun Chen and David E. Carlson and Lawrence Carin},
  title = {{Dynamic Embedding on Textual Networks via a Gaussian Process}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=yOnxXFqJqUN},
}
Figure