Improving Textual Network Embedding with Global Attention via Optimal Transport

Liqun Chen, Guoyin Wang, Chenyang Tao, Dinghan Shen, Pengyu Cheng, Xinyuan Zhang, Wenlin Wang, Yizhe Zhang, Lawrence Carin

Abstract

This paper introduces GANE (Global Attention Network Embedding), a framework designed to improve node representation in textual networks by integrating associated text through a context-aware mechanism. By leveraging Optimal Transport (OT) for sparse attention and a convolutional "attention parsing" module, GANE effectively filters noise and captures complex relational patterns. Experimental results on link prediction and node classification tasks demonstrate that GANE significantly outperforms existing state-of-the-art methods, particularly in data-scarce scenarios.

Introduction

In real-world networks such as citation indices or social media platforms, nodes are often accompanied by rich textual information (e.g., abstracts or user profiles). Traditional network embedding methods typically assign a static vector to each node, which fails to account for the context-dependent nature of node interactions. For example, a researcher's persona may vary depending on whether they are interacting with the "Machine Learning" or "Biology" community.

Existing "context-aware" models utilize mutual attention to highlight relevant text during interactions. However, these approaches suffer from three major drawbacks:

  1. Noise: Standard dense attention matches every word in one document with every word in another, creating significant background noise.
  2. Lack of Structure: Most models rely on simple word-to-word matching without considering higher-level phrases or patterns.
  3. Global Context: They struggle to capture long-range interactions across entire sequences.

Methodology

The authors propose GANE, which represents node embeddings as a combination of a fixed topological vector and a dynamic semantic vector. The framework introduces two novel components:

  • Optimal-Transport-Based Matching: Instead of standard Softmax-based attention, GANE uses Optimal Transport (OT) to align text sequences. By treating word sequences as probability distributions, the model identifies the most efficient alignment, which naturally results in a sparse attention matrix. This sparsity serves as an inherent noise filter, focusing only on the most relevant textual features.
  • Attention Parsing: The model treats the raw attention matrix as a 2D image and applies a Convolutional Neural Network (CNN) to it. This allows the model to recognize higher-order structures, such as matching phrases, rather than just individual word similarities.
The GANE architecture utilizing Optimal Transport for sequence alignment and attention parsing.

The framework is grounded in Reproducing Kernel Hilbert Spaces (RKHS), providing a robust theoretical foundation for the proposed architecture.

Results and Discussion

GANE was evaluated on three standard textual network benchmarks: Cora, Hepth, and Zhihu.

Sparsity and Noise Reduction

A key insight of the study is that sparsity is strength. In textual networks, most words are irrelevant to a specific link. By using Optimal Transport, GANE "forces" the attention to be selective. Visualizations demonstrate that while previous methods like WANE produce blurry, dense attention maps, GANE produces sharp, focused points of alignment.

Comparison of attention maps between WANE (left) and GANE (right), demonstrating the sparsity achieved via Optimal Transport.

Performance Metrics

  • Link Prediction: GANE consistently outperformed state-of-the-art models. On the Cora dataset with only 15% training edges, GANE achieved an AUC of 94.0, compared to CANE’s 86.8.
  • Node Classification: In multi-label classification on Cora, GANE-AP (with Attention Parsing) achieved a Macro-F1 of 82.3 with only 10% of labels known.
  • Robustness: GANE shows its greatest lead over competitors when training data is sparse, suggesting that its global and sparse attention mechanism is highly sample-efficient.

Conclusion

GANE addresses the limitations of traditional textual network embeddings by introducing Optimal Transport for sparse matching and CNNs for attention parsing. While the computational complexity of OT remains a consideration, the model's ability to filter noise and capture structural patterns leads to significant performance gains. Future work may explore extending this framework to other forms of context, such as temporal or geographical data.

Citation: Liqun Chen et al. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

Summary updated

BibTeX
@inproceedings{chen2019improving,
  author = {Liqun Chen and Guoyin Wang and Chenyang Tao and Dinghan Shen and Pengyu Cheng and Xinyuan Zhang and Wenlin Wang and Yizhe Zhang and Lawrence Carin},
  title = {{Improving Textual Network Embedding with Global Attention via Optimal Transport}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=3eEHsLEobaZ},
}
Figure