Network Representation Learning with Rich Text Information

Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun, Edward Y. Chang

Introduction

Network Representation Learning (NRL) is a cornerstone of modern graph analysis, transforming nodes into low-dimensional vectors for tasks like classification and link prediction. While random-walk-based methods like DeepWalk have seen widespread adoption, they often ignore the rich text information associated with nodes, leading to suboptimal performance in sparse or noisy networks. This paper introduces Text-Associated DeepWalk (TADW), a framework that bridges the gap between structural graph embeddings and semantic text features.

Theoretical Foundation: DeepWalk as Matrix Factorization

A pivotal contribution of this work is the mathematical proof that DeepWalk’s skip-gram objective is equivalent to factorizing a specific matrix $M$. In this matrix, each entry $M_{ij}$ represents the logarithm of the average probability that a random walk starting at vertex $v_i$ reaches $v_j$ within a fixed window. This insight allows researchers to move beyond "black box" neural walks and apply well-understood optimization techniques from matrix theory to graph embeddings.

The TADW Framework

Building on the matrix factorization perspective, the authors propose TADW. Unlike standard DeepWalk, which factorizes $M$ into two latent matrices ($M \approx W^T H$), TADW incorporates a pre-computed text feature matrix $T$ (derived from TF-IDF and reduced via SVD) directly into the objective function.

Comparison of Matrix Factorization frameworks for DeepWalk (a) and TADW (b).

By defining the factorization as $M \approx W^T H T$, TADW treats text as a fundamental component of the learning process. In this setup, $T$ is fixed, while $W$ and $H$ are learned. The final node representation is formed by concatenating $W$ and $HT$, effectively forcing structural embeddings to be consistent with the underlying semantic content.

Experimental Evaluation

The framework was evaluated on three benchmark citation and document networks:

  • Cora: 2,708 machine learning papers across 7 classes.
  • Citeseer: 3,312 publications across 6 classes.
  • Wiki: 2,405 Wikipedia articles across 19 classes.

Key Results and Insights

TADW consistently demonstrated superiority over structure-only methods (DeepWalk) and content-only methods (PLSA), particularly in challenging scenarios.

Classification accuracy on Cora, Citeseer, and Wiki datasets across varying values of the parameter Lambda and embedding dimensions k.
  • Performance in Sparse Data: TADW's most significant gains occur when labeled data is scarce. With only 1% of nodes labeled, TADW outperformed baselines by 5% to 20% in classification accuracy.
  • Robustness to Noise: In the Citeseer dataset, which is notoriously noisy, TADW's joint modeling proved far superior to the "naive concatenation" of features, proving that integrated modeling captures the complex interplay between structure and content.
  • Parameter Stability: As shown in the experimental results, the model maintains stable accuracy across various values of the harmonic parameter $\lambda$ and embedding dimensions $k$, indicating its reliability across different graph types and scales.

Conclusion

TADW demonstrates that integrating rich text information through an inductive matrix completion framework significantly enhances network representations. By leveraging the mathematical equivalence between skip-grams and matrix factorization, the authors provide a principled way to incorporate auxiliary node features, paving the way for more robust and semantically aware graph mining in real-world applications.

Summary updated

Abstract

Representation learning has shown its effectiveness in many tasks such as image classification and text mining. Network representation learning aims at learning distributed vector representation for each vertex in a network, which is also increasingly recognized as an important aspect for network analysis. Most network representation learning methods investigate network structures for learning. In reality, network vertices contain rich information (such as text), which cannot be well applied with algorithmic frameworks of typical representation learning methods. By proving that DeepWalk, a state-of-the-art network representation method, is actually equivalent to matrix factorization (MF), we propose text-associated DeepWalk (TADW). TADW incorporates text features of vertices into network representation learning under the framework of matrix factorization. We evaluate our method and various baseline methods by applying them to the task of multi-class classification of vertices. The experimental results show that, our method outperforms other baselines on all three datasets, especially when networks are noisy and training ratio is small. The source code of this paper can be obtained from https://github.com/albertyang33/TADW.
BibTeX
@inproceedings{yang2015network,
  author = {Cheng Yang and Zhiyuan Liu and Deli Zhao and Maosong Sun and Edward Y. Chang},
  title = {{Network Representation Learning with Rich Text Information}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure