Abstract
Textual network embedding aims to learn low-dimensional representations of text-annotated nodes in a graph. Prior work in this area has typically focused on fixed graph structures; however, real-world networks are often dynamic. We address this challenge with a novel end-to-end node-embedding model, called Dynamic Embedding for Textual Networks with a Gaussian Process (DetGP). After training, DetGP can be applied efficiently to dynamic graphs without re-training or backpropagation. The learned representation of each node is a combination of textual and structural embeddings. Because the structure is allowed to be dynamic, our method uses the Gaussian process to take advantage of its non-parametric properties. To use both local and global graph structures, diffusion is used to model multiple hops between neighbors. The relative importance of global versus local structure for the embeddings is learned automatically. With the non-parametric nature of the Gaussian process, updating the embeddings for a changed graph structure requires only a forward pass through the learned model. Considering link prediction and node classification, experiments demonstrate the empirical effectiveness of our method compared to baseline approaches. We further show that DetGP can be straightforwardly and efficiently applied to dynamic textual networks.
Introduction
Textual networks, where nodes are annotated with rich textual information, are fundamental to modern data science, appearing in citation graphs, social media, and knowledge bases. While many embedding techniques exist, they typically struggle with the dynamic nature of real-world networks. Adding a single node or edge often necessitates retraining the entire model, which is computationally prohibitive in large-scale environments. To address this, we introduce DetGP (Dynamic Embedding for Textual Networks with a Gaussian Process), a framework that leverages the non-parametric power of Gaussian Processes to enable efficient, inductive updates.
The DetGP Framework
DetGP is an end-to-end model that jointly learns from textual content and graph topology. The architecture consists of several key components:
- Textual Encoding: Raw text is transformed into a latent embedding ($x_n$) using an encoder, such as a word-averaging layer.
- Gaussian Process Structural Layer: This core component maps textual features to structural embeddings ($s_n$). By treating this mapping as a Gaussian Process, the model learns a distribution over functions, providing a robust bridge between content and context.
- Multi-hop Diffusion: To capture global structure beyond immediate neighbors, DetGP employs a multi-hop transition matrix. The model automatically learns the optimal weighting for different neighborhood scales (e.g., 1-hop vs. 2-hop).
- Inducing Points for Scalability: To overcome the $O(N^3)$ complexity of standard GPs, DetGP utilizes a small set of "inducing points." These act as landmarks in the embedding space, allowing the model to scale to large datasets like DBLP.
When a new node arrives, the model simply calculates its similarity to the fixed inducing points and its new neighbors. This requires only a forward pass, making it orders of magnitude faster than retraining.
Experimental Evaluation
The model was rigorously tested on three benchmark citation networks: Cora, DBLP, and HepTh. We compared DetGP against a variety of state-of-the-art baselines, including TADW, CANE, and WANE.
Performance and Efficiency
As shown in the results below, DetGP (specifically the DWavg variant) consistently achieves superior performance in node classification and link prediction tasks.
A key advantage of DetGP is its performance in dynamic settings. In tests where nodes were added after training, DetGP significantly outperformed inductive baselines like GraphSAGE. For link prediction on Cora with only 10% of nodes seen during training, DetGP achieved an AUC of 62.9, compared to GraphSAGE’s 62.2 and a simple text-only baseline of 61.2. The use of inducing points allowed the model to handle the DBLP dataset (over 60k nodes) effectively, a scale that typically breaks standard GP implementations.
Discussion and Future Directions
DetGP's success stems from its use of the GP as a functional mapping. This allows the model to "predict" a new node's structural position based on its text and local connections. Furthermore, the automatic smoothing provided by multi-hop diffusion ensures the model adapts to the specific connectivity patterns of each dataset.
Future work may involve exploring more complex kernels (e.g., RBF) to capture non-linear relationships, or extending the model to handle temporal evolution where node text changes over time. Additionally, optimizing the selection of inducing points could further enhance performance on massive, complex networks.
Summary updated