Natural language is inherently dual-natured. On the surface, it presents itself as a linear sequence of tokens flowing chronologically from left to right. Beneath this surface lies a rich, non-linear hierarchical structure—dictated by syntactic dependency trees, semantic frameworks like Abstract Meaning Representation (AMR), or high-level discourse relations.
For years, NLP models leaned heavily toward one side of this duality. Purely sequential models (like LSTMs or standard Transformers) excel at capturing local word order and temporal context, but they can struggle to track long-range dependencies or explicit grammatical relationships when sentences become complex. Conversely, purely graph-based models (like Graph Convolutional Networks) are excellent at "teleporting" information across structural shortcuts, but they inherently discard the linear topology and word order of the text, leading to information loss.
To bridge this gap, a prominent research direction has emerged: hybrid sequential-structural encoding. By fusing sequence-based and graph-based representations, these architectures leverage the complementary strengths of both paradigms to build a more comprehensive understanding of text.
Stacked Architectures: Passing the Baton
One of the most intuitive ways to combine these two worlds is to stack them sequentially. Typically, a bidirectional sequence model processes the raw text first to capture local context and word order, and its outputs are then passed into a graph neural network layer that operates on a structural graph (such as a dependency tree).
For instance, Thien Huu Nguyen et al. in Graph Convolutional Networks With Argument-Aware Pooling for Event Detection introduced an architecture that integrates bidirectional recurrent units with graph convolutional networks (GCNs). In this setup, the sequential layer captures the global flow of the text, while the subsequent graph layer processes the hierarchical grammatical relationships.
Similarly, Diego Marcheggiani et al. applied this stacked approach to semantic role labeling in Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling, stacking graph convolutional layers directly on top of bidirectional recurrent layers. This allows the structural layer to build upon a sequence-aware foundation.
As demonstrated by Yuhao Zhang et al. in Graph Convolution over Pruned Dependency Trees Improves Relation Extraction, this pipeline ensures that the system captures local word order alongside structural dependencies, compensating for the tendency of graph-only models to overlook the importance of linear context in natural language.
[Raw Text] ---> [Sequential Layer (BiLSTM/Transformer)] ---> [Graph Layer (GCN/GAT)] ---> [Structural Embeddings]
Parallel and Dual-Stream Fusion
Instead of processing features sequentially, another prominent paradigm runs sequence and graph encoders in parallel, fusing their representations later in the network. This ensures that neither representation is bottlenecked or distorted by the other before integration.
In the single-author work Improving Neural Machine Translation with the Abstract Meaning Representation by Combining Graph and Sequence Transformers, Changmao Li proposes a dual-stream encoding architecture that runs a standard sequence transformer in parallel with a graph-based transformer. This approach allows the system to capture surface-level linguistic patterns and deep structural semantic relationships simultaneously, significantly reducing translation errors like word repetition and hallucinations.
In a similar vein, Linxiao Wu et al. developed a framework in Graph Neural Network Framework for Sentiment Analysis Using Syntactic Feature that processes text through a bidirectional sequential layer and a graph-based structural layer in parallel. For long-form documents like movie scripts, Maitreya Prafulla Chitale et al. designed a late-fusion architecture in DiscoGraMS: Enhancing Movie Screen-Play Summarization using Movie Character-Aware Discourse Graph that merges the outputs of a transformer encoder and a graph attention network to capture both local nuances and long-range character-discourse dependencies.
Deep Integration: Gated Cells and Graph-as-Token
Beyond simple stacking or parallel concatenation, researchers have designed tighter integrations that embed structural awareness directly into recurrent cells or token sequences.
Cell-Level Gating
To prevent noisy graph structures from degrading sequential learning, Lu Xu et al. proposed the Synergized-LSTM (Syn-LSTM) in Better Feature Integration for Named Entity Recognition. They introduced an additional gating mechanism inside the recurrent cell to independently regulate the flow of information from sequential word embeddings and graph-encoded structural features. This allows the model to selectively integrate local context and long-range dependencies at each individual time step.
Graph-as-Token
To completely bypass the architectural complexity and training instability of hybrid GNN-Transformer systems, Cunxiang Wang et al. introduced the "Graph-as-Token" (GST) method in Exploiting Abstract Meaning Representation for Open-Domain Question Answering. Instead of using a separate graph network, they appended graph-derived nodes and edges as special tokens directly to the text token sequence. This creates a single unified input, allowing a standard pre-trained transformer to natively learn sequential-structural interactions through its self-attention mechanism.
Why It Works: The Intuition of Structural Shortcuts
To understand why this hybrid approach consistently outperforms pure sequence or pure graph models, consider a sentence where a subject and its verb are separated by a long, winding relative clause:
"The man who was walking down the street while talking on his phone and carrying groceries tripped."
A sequential model must pass hidden states through every single intervening word ("who", "was", "walking", etc.), risking information decay or dilution over long distances. The model might lose track of the fact that "man" is the entity that "tripped."
A graph model operating on a dependency tree, however, creates a direct edge—a structural shortcut—between "man" and "tripped".
+------------------ structural shortcut -----------------+
| v
[ The man ] -> [ who ] -> [ was ] -> ... -> [ groceries ] -> [ tripped ]
By combining these two representations, the hybrid model retains the local word order (knowing that "street" comes before "phone") while simultaneously leveraging the structural shortcut to instantly bind the subject to its action. This dual-pathway approach ensures that the final embeddings capture both the immediate textual context and the underlying logical framework of the input text.