Tri-Party Deep Network Representation

Shirui Pan, Jia Wu, Xingquan Zhu, Chengqi Zhang, Yang Wang

Introduction

Network representation learning aims to transform complex graph nodes into low-dimensional vectors (embeddings) suitable for downstream machine learning tasks. Traditional methods often focus exclusively on network structure or node content in isolation, creating information silos. TriDNR (Tri-Party Deep Network Representation) addresses these limitations by jointly learning node representations from three distinct sources: network structure (links), node content (text), and node labels.

Overview of the TriDNR framework: mapping a network with structure, text, and labels to a clustered representation.

By coupling these "tri-party" signals into a unified deep learning architecture, TriDNR leverages available labels as anchors to guide the embedding process. This semi-supervised approach significantly improves performance in tasks like node classification by ensuring that representations are sensitive to both the topology and the semantics of the graph.

Methodology

TriDNR utilizes a coupled neural network architecture based on the Skip-gram model. The learning process integrates information at three levels simultaneously:

  1. Network Structure Level: Using random walks, the graph is treated as a collection of "sentences" where nodes are "words." The model maximizes the probability of observing a node's neighbors to capture local topology.
  2. Node Content Level: Text associated with a node (e.g., a paper title) is treated as a document. The model learns a node vector optimized to predict the specific words within that node's content, capturing the correlation between nodes and vocabulary.
  3. Node Label Level: For labeled nodes, a "class vector" is learned that also predicts the words associated with that class. This forces node embeddings to align with the underlying categorical structure.
Architectural comparison between the standard DeepWalk model and the proposed TriDNR model.

The model is optimized using Hierarchical Softmax, ensuring scalability to large networks by reducing computational complexity in the output layer. Unlike previous matrix factorization methods, TriDNR's neural approach captures word order and context more effectively while scaling linearly with the number of words.

Experimental Evaluation

The authors evaluated TriDNR on two major citation network benchmarks: DBLP (over 60,000 papers across 4 classes) and CiteSeer-M10 (approximately 10,000 publications across 10 classes). In these datasets, paper titles served as node content, and research areas served as labels.

Key Results

TriDNR consistently outperformed state-of-the-art competitors, including DeepWalk, LINE, and Doc2Vec. On the CiteSeer-M10 dataset, TriDNR achieved a Macro F1 score of 0.777, a significant improvement over DeepWalk's 0.434 (approximately a 79% relative gain) and the previous state-of-the-art TADW (0.681).

Visual analysis using t-SNE confirmed that TriDNR produces cleaner, more distinct clusters compared to unsupervised methods. This suggests that even a small amount of label information acts as a powerful anchor, guiding the embedding process to better reflect class boundaries. The model also proved robust across different embedding dimensions, ranging from 50 to 300.

Discussion and Future Work

The success of TriDNR highlights the synergy between structure and content: structure compensates for sparse text, while content aids when links are few. However, the model's reliance on text content suggests potential limitations in networks where node features are non-existent or non-textual. Additionally, the model introduces a weight parameter (α) to balance structure versus content, which may require manual tuning for new datasets. Future research directions include extending the framework to heterogeneous networks and incorporating more complex node features beyond simple vocabulary.

Summary updated

Abstract

Information network mining often requires examination of linkage relationships between nodes for analysis. Recently, network representation has emerged to represent each node in a vector format, embedding network structure, so off-the-shelf machine learning methods can be directly applied for analysis. To date, existing methods only focus on one aspect of node information and cannot leverage node labels. In this paper, we propose TriDNR, a tri-party deep network representation model, using information from three parties: node structure, node content, and node labels (if available) to jointly learn optimal node representation. TriDNR is based on our new coupled deep natural language module, whose learning is enforced at three levels: (1) at the network structure level, TriDNR exploits inter-node relationship by maximizing the probability of observing surrounding nodes given a node in random walks; (2) at the node content level, TriDNR captures node-word correlation by maximizing the co-occurrence of word sequence given a node; and (3) at the node label level, TriDNR models label-word correspondence by maximizing the probability of word sequence given a class label. The tri-party information is jointly fed into the neural network model to mutually enhance each other to learn optimal representation, and results in up to 79% classification accuracy gain, compared to state-of-the-art methods.
BibTeX
@inproceedings{pan2016tri,
  author = {Shirui Pan and Jia Wu and Xingquan Zhu and Chengqi Zhang and Yang Wang},
  title = {{Tri-Party Deep Network Representation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure