Improving Textual Network Embedding with Global Attention via Optimal Transport

Liqun Chen, Guoyin Wang, Chenyang Tao, Dinghan Shen, Pengyu Cheng, Xinyuan Zhang, Wenlin Wang, Yizhe Zhang, Lawrence Carin

Introduction

Network embedding is a fundamental task in data science, aimed at mapping nodes into low-dimensional latent spaces for downstream applications like link prediction and node classification. While traditional methods focus on graph topology, modern "textual networks"—such as citation graphs or social media platforms—contain rich natural language data associated with each node. Effectively integrating this textual information remains a challenge. Existing context-aware models often rely on dense attention mechanisms that are prone to noise and fail to capture long-range relational dependencies.

In this work, we introduce the Global Attention Network Embedding (GANE) framework. GANE addresses these limitations by leveraging Optimal Transport (OT) to produce sparse, content-aware attention and a convolutional "parsing" module to extract high-level relational features.

Methodology: The GANE Framework

The core innovation of GANE lies in how it aligns the text of two interacting nodes to create a context-aware embedding. Instead of static representations, the embedding of a node $u$ is dynamically adjusted based on the node $v$ it is being compared to.

The GANE architecture, highlighting the Optimal Transport module and the generation of the transport matrix T.

Sparse Attention via Optimal Transport

Traditional attention mechanisms, like Softmax, typically produce dense matrices where every word in one sequence is matched to every word in another, often incorporating irrelevant "background noise." GANE treats text sequences as probability distributions in a word-vector space and utilizes Optimal Transport to find the most efficient alignment between them.

By solving the OT problem (using the efficient Sinkhorn algorithm), the model generates a transport matrix $T$. This matrix is naturally sparse, focusing only on the most semantically relevant word pairs. This sparsity acts as an inherent noise filter, ensuring that the resulting embedding captures the "gist" of the relationship rather than getting lost in common, non-informative words.

Comparison of attention matrices: WANE (left) shows dense, noisy matching, while GANE (right) produces a sparse, focused transport plan.

High-Level Attention Parsing

To further improve relational inference, GANE treats the transport matrix $T$ as a 2D image and applies a Convolutional Neural Network (CNN). This "Attention Parsing" (AP) module allows the model to recognize spatial patterns within the alignment. For instance, it can understand that a specific keyword in one node relates to a broader phrase or context in another, rather than requiring exact word-to-word matches.

Experimental Results

We evaluated GANE on three benchmark datasets: Cora (machine learning citations), Hepth (physics citations), and Zhihu (social network interactions). We tested two variants: GANE-OT (basic optimal transport) and GANE-AP (with the attention parsing module).

  • Link Prediction: GANE-AP achieved state-of-the-art results, reaching an AUC of 99.3% on the Cora dataset. Notably, GANE showed significant robustness in low-data scenarios. On the Zhihu dataset, with only 15% training edges, GANE-AP outperformed the previous best model (WANE) by nearly 6 percentage points (64.6% vs. 58.7% AUC).
  • Node Classification: In multi-label classification tasks, GANE consistently outperformed competitors across various training label ratios, demonstrating that its context-aware embeddings are highly informative for downstream tasks.

Conclusion

The GANE framework represents a significant step forward in textual network embedding. By replacing dense attention with sparse Optimal Transport and introducing a convolutional parsing module, the model effectively filters noise and captures complex semantic relationships. These innovations lead to more precise and robust node representations, particularly in challenging, low-data environments.

Summary updated

Abstract

Constituting highly informative network embeddings is an important tool for network analysis. It encodes network topology, along with other useful side information, into low-dimensional node-based feature representations that can be exploited by statistical modeling. This work focuses on learning context-aware network embeddings augmented with text data. We reformulate the network-embedding problem, and present two novel strategies to improve over traditional attention mechanisms: ($i$) a content-aware sparse attention module based on optimal transport, and ($ii$) a high-level attention parsing module. Our approach yields naturally sparse and self-normalized relational inference. It can capture long-term interactions between sequences, thus addressing the challenges faced by existing textual network embedding schemes. Extensive experiments are conducted to demonstrate our model can consistently outperform alternative state-of-the-art methods.
BibTeX
@inproceedings{chen2019improving,
  author = {Liqun Chen and Guoyin Wang and Chenyang Tao and Dinghan Shen and Pengyu Cheng and Xinyuan Zhang and Wenlin Wang and Yizhe Zhang and Lawrence Carin},
  title = {{Improving Textual Network Embedding with Global Attention via Optimal Transport}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=3eEHsLEobaZ},
}
Figure