Linked Recurrent Neural Networks

Zhiwei Wang, Yao Ma, Dawei Yin, Jiliang TangarXiv 2018 · arXiv

Introduction

Traditional Recurrent Neural Networks (RNNs) typically operate under the assumption that data sequences are independent and identically distributed (i.i.d.). However, in many real-world applications, sequences are explicitly connected. For instance, web pages are linked via hyperlinks, scientific papers are connected through citations, and biological sequences like DNA or proteins interact within complex regulatory networks.

When sequences are linked, they often exhibit homophily—the tendency for connected entities to share similar characteristics or topics. Standard RNNs, while effective at capturing internal sequential patterns, are often "blind" to these external relationships. This leads to a "double whammy": models struggle because linked sequences violate the i.i.d. assumption, and they fail to leverage the rich relational information that could improve predictive accuracy. To address this, the authors propose LinkedRNN, a framework designed to model sequences that are both internally structured and externally linked.

The LinkedRNN Architecture

The LinkedRNN framework integrates sequential modeling with graph-based relational learning by stacking two specialized layers to process data.

The LinkedRNN architecture, showcasing the integration of the RNN Layer for intra-sequence patterns and the Link Layer for inter-sequence relationships.

1. The RNN Layer (Intra-sequence Information)

This layer focuses on the internal dynamics of each individual sequence. It utilizes Gated Recurrent Units (GRUs) to process sequences. To transform the resulting hidden states into a fixed-length vector representation, the authors explored two primary aggregation methods:

  • Last State: Using the final hidden state of the GRU.
  • Attention Mechanism: Weighing different parts of the sequence based on their relevance to the overall context.

The Link Layer introduces the relational component. It takes the sequence representations from the RNN layer and performs graph-based aggregation. Using an adjacency matrix representing the links between sequences, this layer pulls in information from a sequence's "neighbors." This process can be stacked over $M$ layers, allowing the model to capture higher-order relationships (neighbors-of-neighbors).

Finally, the model aggregates representations from different depths of the link layer before passing them to a task-specific head for classification or regression.

Experimental Evaluation

The model was evaluated on two diverse real-world datasets to test its versatility:

  • DBLP (Citation Network): This dataset consists of approximately 47,000 paper abstracts. Here, the sequences are word embeddings of the text, and the links represent citations between papers. The objective is venue classification.
  • BOOHEE (Social/Health Network): This dataset includes data from roughly 18,000 users of a weight-management application. The sequences are weekly weight logs, and the links are social interactions (follows, comments, mentions). The task is a regression problem: predicting a user's future weight.

Key Insights and Analysis

The experimental results provided several critical insights into the behavior of interconnected sequences:

  • Complementary Information: Models relying solely on link information (e.g., GCN, node2vec) or solely on sequential content (vanilla RNN) were consistently outperformed by LinkedRNN. This confirms that links provide a distinct signal that is not fully captured by the content alone.
  • The Role of Attention: Using an attention mechanism to aggregate RNN hidden states proved superior to simply using the last hidden state. This suggests that when sequences are linked, specific "anchor points" within the content are particularly important for defining those relationships.
  • Optimal Network Depth: A parameter analysis revealed a "sweet spot" for the number of link layers.
Impact of the number of link layers on model performance, highlighting the optimal "sweet spot" at two layers.

As shown in the analysis, performance typically peaks at approximately two layers. While one layer is insufficient to capture the broader network structure, exceeding three layers tends to introduce noise or lead to "over-smoothing," where the unique signal of the original sequence is diluted by excessive neighbor data.

Results and Conclusion

LinkedRNN demonstrated significant performance gains over state-of-the-art baselines:

  • DBLP: Achieved a Micro-F1 score of 0.8531, compared to 0.8025 for a standard RNN and 0.7180 for a GCN.
  • BOOHEE: Achieved a Mean Squared Error (MSE) of 6.3517, outperforming the vanilla RNN (6.8033) and the social-only node2vec baseline (7.0390).

The success of LinkedRNN highlights the importance of "coherent modeling." Simple "late fusion" techniques—such as concatenating the outputs of an RNN and a GCN—were less effective than the integrated approach of LinkedRNN.

Future Directions

While LinkedRNN represents a significant step forward, several avenues for future research remain. The current model treats links as unweighted and undirected; future iterations could incorporate link "strength" or directionality (e.g., the directed nature of citations). Additionally, extending the framework to handle dynamic, evolving networks and inductive learning for entirely new sequences are promising areas for expansion.

Summary updated

Abstract

Recurrent Neural Networks (RNNs) have been proven to be effective in modeling sequential data and they have been applied to boost a variety of tasks such as document classification, speech recognition and machine translation. Most of existing RNN models have been designed for sequences assumed to be identically and independently distributed (i.i.d). However, in many real-world applications, sequences are naturally linked. For example, web documents are connected by hyperlinks; and genes interact with each other. On the one hand, linked sequences are inherently not i.i.d., which poses tremendous challenges to existing RNN models. On the other hand, linked sequences offer link information in addition to the sequential information, which enables unprecedented opportunities to build advanced RNN models. In this paper, we study the problem of RNN for linked sequences. In particular, we introduce a principled approach to capture link information and propose a linked Recurrent Neural Network (LinkedRNN), which models sequential and link information coherently. We conduct experiments on real-world datasets from multiple domains and the experimental results validate the effectiveness of the proposed framework.
BibTeX
@misc{wang2018linked,
  author = {Zhiwei Wang and Yao Ma and Dawei Yin and Jiliang Tang},
  title = {{Linked Recurrent Neural Networks}},
  year = {2018},
  eprint = {1808.06170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.06170},
}
Figure