Introduction
Network Representation Learning (NRL) is a cornerstone of modern graph analysis, transforming nodes into low-dimensional vectors for tasks like classification and link prediction. While random-walk-based methods like DeepWalk have seen widespread adoption, they often ignore the rich text information associated with nodes, leading to suboptimal performance in sparse or noisy networks. This paper introduces Text-Associated DeepWalk (TADW), a framework that bridges the gap between structural graph embeddings and semantic text features.
Theoretical Foundation: DeepWalk as Matrix Factorization
A pivotal contribution of this work is the mathematical proof that DeepWalk’s skip-gram objective is equivalent to factorizing a specific matrix $M$. In this matrix, each entry $M_{ij}$ represents the logarithm of the average probability that a random walk starting at vertex $v_i$ reaches $v_j$ within a fixed window. This insight allows researchers to move beyond "black box" neural walks and apply well-understood optimization techniques from matrix theory to graph embeddings.
The TADW Framework
Building on the matrix factorization perspective, the authors propose TADW. Unlike standard DeepWalk, which factorizes $M$ into two latent matrices ($M \approx W^T H$), TADW incorporates a pre-computed text feature matrix $T$ (derived from TF-IDF and reduced via SVD) directly into the objective function.
By defining the factorization as $M \approx W^T H T$, TADW treats text as a fundamental component of the learning process. In this setup, $T$ is fixed, while $W$ and $H$ are learned. The final node representation is formed by concatenating $W$ and $HT$, effectively forcing structural embeddings to be consistent with the underlying semantic content.
Experimental Evaluation
The framework was evaluated on three benchmark citation and document networks:
- Cora: 2,708 machine learning papers across 7 classes.
- Citeseer: 3,312 publications across 6 classes.
- Wiki: 2,405 Wikipedia articles across 19 classes.
Key Results and Insights
TADW consistently demonstrated superiority over structure-only methods (DeepWalk) and content-only methods (PLSA), particularly in challenging scenarios.
- Performance in Sparse Data: TADW's most significant gains occur when labeled data is scarce. With only 1% of nodes labeled, TADW outperformed baselines by 5% to 20% in classification accuracy.
- Robustness to Noise: In the Citeseer dataset, which is notoriously noisy, TADW's joint modeling proved far superior to the "naive concatenation" of features, proving that integrated modeling captures the complex interplay between structure and content.
- Parameter Stability: As shown in the experimental results, the model maintains stable accuracy across various values of the harmonic parameter $\lambda$ and embedding dimensions $k$, indicating its reliability across different graph types and scales.
Conclusion
TADW demonstrates that integrating rich text information through an inductive matrix completion framework significantly enhances network representations. By leveraging the mathematical equivalence between skip-grams and matrix factorization, the authors provide a principled way to incorporate auxiliary node features, paving the way for more robust and semantically aware graph mining in real-world applications.
Summary updated