Introduction
Traditional Recurrent Neural Networks (RNNs) typically operate under the assumption that data sequences are independent and identically distributed (i.i.d.). However, in many real-world applications, sequences are explicitly connected. For instance, web pages are linked via hyperlinks, scientific papers are connected through citations, and biological sequences like DNA or proteins interact within complex regulatory networks.
When sequences are linked, they often exhibit homophily—the tendency for connected entities to share similar characteristics or topics. Standard RNNs, while effective at capturing internal sequential patterns, are often "blind" to these external relationships. This leads to a "double whammy": models struggle because linked sequences violate the i.i.d. assumption, and they fail to leverage the rich relational information that could improve predictive accuracy. To address this, the authors propose LinkedRNN, a framework designed to model sequences that are both internally structured and externally linked.
The LinkedRNN Architecture
The LinkedRNN framework integrates sequential modeling with graph-based relational learning by stacking two specialized layers to process data.
1. The RNN Layer (Intra-sequence Information)
This layer focuses on the internal dynamics of each individual sequence. It utilizes Gated Recurrent Units (GRUs) to process sequences. To transform the resulting hidden states into a fixed-length vector representation, the authors explored two primary aggregation methods:
- Last State: Using the final hidden state of the GRU.
- Attention Mechanism: Weighing different parts of the sequence based on their relevance to the overall context.
2. The Link Layer (Inter-sequence Information)
The Link Layer introduces the relational component. It takes the sequence representations from the RNN layer and performs graph-based aggregation. Using an adjacency matrix representing the links between sequences, this layer pulls in information from a sequence's "neighbors." This process can be stacked over $M$ layers, allowing the model to capture higher-order relationships (neighbors-of-neighbors).
Finally, the model aggregates representations from different depths of the link layer before passing them to a task-specific head for classification or regression.
Experimental Evaluation
The model was evaluated on two diverse real-world datasets to test its versatility:
- DBLP (Citation Network): This dataset consists of approximately 47,000 paper abstracts. Here, the sequences are word embeddings of the text, and the links represent citations between papers. The objective is venue classification.
- BOOHEE (Social/Health Network): This dataset includes data from roughly 18,000 users of a weight-management application. The sequences are weekly weight logs, and the links are social interactions (follows, comments, mentions). The task is a regression problem: predicting a user's future weight.
Key Insights and Analysis
The experimental results provided several critical insights into the behavior of interconnected sequences:
- Complementary Information: Models relying solely on link information (e.g., GCN, node2vec) or solely on sequential content (vanilla RNN) were consistently outperformed by LinkedRNN. This confirms that links provide a distinct signal that is not fully captured by the content alone.
- The Role of Attention: Using an attention mechanism to aggregate RNN hidden states proved superior to simply using the last hidden state. This suggests that when sequences are linked, specific "anchor points" within the content are particularly important for defining those relationships.
- Optimal Network Depth: A parameter analysis revealed a "sweet spot" for the number of link layers.
As shown in the analysis, performance typically peaks at approximately two layers. While one layer is insufficient to capture the broader network structure, exceeding three layers tends to introduce noise or lead to "over-smoothing," where the unique signal of the original sequence is diluted by excessive neighbor data.
Results and Conclusion
LinkedRNN demonstrated significant performance gains over state-of-the-art baselines:
- DBLP: Achieved a Micro-F1 score of 0.8531, compared to 0.8025 for a standard RNN and 0.7180 for a GCN.
- BOOHEE: Achieved a Mean Squared Error (MSE) of 6.3517, outperforming the vanilla RNN (6.8033) and the social-only node2vec baseline (7.0390).
The success of LinkedRNN highlights the importance of "coherent modeling." Simple "late fusion" techniques—such as concatenating the outputs of an RNN and a GCN—were less effective than the integrated approach of LinkedRNN.
Future Directions
While LinkedRNN represents a significant step forward, several avenues for future research remain. The current model treats links as unweighted and undirected; future iterations could incorporate link "strength" or directionality (e.g., the directed nature of citations). Additionally, extending the framework to handle dynamic, evolving networks and inductive learning for entirely new sequences are promising areas for expansion.
Summary updated