Introduction
Network embedding is a fundamental task in data science, aimed at mapping nodes into low-dimensional latent spaces for downstream applications like link prediction and node classification. While traditional methods focus on graph topology, modern "textual networks"—such as citation graphs or social media platforms—contain rich natural language data associated with each node. Effectively integrating this textual information remains a challenge. Existing context-aware models often rely on dense attention mechanisms that are prone to noise and fail to capture long-range relational dependencies.
In this work, we introduce the Global Attention Network Embedding (GANE) framework. GANE addresses these limitations by leveraging Optimal Transport (OT) to produce sparse, content-aware attention and a convolutional "parsing" module to extract high-level relational features.
Methodology: The GANE Framework
The core innovation of GANE lies in how it aligns the text of two interacting nodes to create a context-aware embedding. Instead of static representations, the embedding of a node $u$ is dynamically adjusted based on the node $v$ it is being compared to.
Sparse Attention via Optimal Transport
Traditional attention mechanisms, like Softmax, typically produce dense matrices where every word in one sequence is matched to every word in another, often incorporating irrelevant "background noise." GANE treats text sequences as probability distributions in a word-vector space and utilizes Optimal Transport to find the most efficient alignment between them.
By solving the OT problem (using the efficient Sinkhorn algorithm), the model generates a transport matrix $T$. This matrix is naturally sparse, focusing only on the most semantically relevant word pairs. This sparsity acts as an inherent noise filter, ensuring that the resulting embedding captures the "gist" of the relationship rather than getting lost in common, non-informative words.
High-Level Attention Parsing
To further improve relational inference, GANE treats the transport matrix $T$ as a 2D image and applies a Convolutional Neural Network (CNN). This "Attention Parsing" (AP) module allows the model to recognize spatial patterns within the alignment. For instance, it can understand that a specific keyword in one node relates to a broader phrase or context in another, rather than requiring exact word-to-word matches.
Experimental Results
We evaluated GANE on three benchmark datasets: Cora (machine learning citations), Hepth (physics citations), and Zhihu (social network interactions). We tested two variants: GANE-OT (basic optimal transport) and GANE-AP (with the attention parsing module).
- Link Prediction: GANE-AP achieved state-of-the-art results, reaching an AUC of 99.3% on the Cora dataset. Notably, GANE showed significant robustness in low-data scenarios. On the Zhihu dataset, with only 15% training edges, GANE-AP outperformed the previous best model (WANE) by nearly 6 percentage points (64.6% vs. 58.7% AUC).
- Node Classification: In multi-label classification tasks, GANE consistently outperformed competitors across various training label ratios, demonstrating that its context-aware embeddings are highly informative for downstream tasks.
Conclusion
The GANE framework represents a significant step forward in textual network embedding. By replacing dense attention with sparse Optimal Transport and introducing a convolutional parsing module, the model effectively filters noise and captures complex semantic relationships. These innovations lead to more precise and robust node representations, particularly in challenging, low-data environments.
Summary updated