Language is inherently dual-natured. On the surface, it is a sequence of tokens where the order of words creates meaning. Beneath that surface lies a complex web of structural relationships—syntactic trees, semantic dependencies, and knowledge graphs—that link concepts across long distances.
While standard Transformers and Recurrent Neural Networks (RNNs) excel at capturing the local "flow" of language, they often struggle to reason over the underlying "skeleton" of information. Conversely, Graph Neural Networks (GNNs) are designed for relational reasoning but can lose the vital nuances of word order. A growing body of research focuses on hybrid graph-sequence architectures that fuse these two modalities to capture both the linear and structural dimensions of data.
The Synergy of Flow and Structure
The core intuition behind this research is that local word proximity and global relational structures provide non-redundant signals. In tasks like Structured Neural Summarization, Patrick Fernandes and colleagues demonstrated that while sequence models handle the "flesh" of the text, GNNs provide the "thread" that prevents the model from losing track of relationships that span hundreds of tokens.
This synergy is particularly effective because it reduces the "dependency distance." In a linear sequence, a subject and its distant object might be separated by dozens of words, making it hard for a model to link them. In a graph—such as a syntactic dependency tree—those same two words might be directly connected by a single edge. By integrating these structures, models can "shortcut" the linear distance, leading to more factually consistent outputs in areas like Grammatical Error Correction and Event Detection.
Architectural Patterns of Fusion
Researchers have converged on several distinct ways to marry these two paradigms:
1. Sequential-then-Structural (Stacked)
The most common approach uses a sequential model (like a BiLSTM or BERT) to generate initial context-aware embeddings, which then serve as node features for a GNN. Yuhao Zhang showed that placing a graph convolutional layer after a bidirectional sequence model allows the system to capture local word order before refining those representations through structural dependencies. This is highly effective for Relation Extraction, where the model must understand both the sentence's tone and the formal links between entities.
2. Parallel Streams (Dual-Encoding)
Some models process text and structure in separate, simultaneous paths. In SinLG, Xuhui Ren uses a Siamese network to merge a Pre-trained Language Model (PLM) with a GNN. Similarly, Bohao Yang employs parallel transformers—one for text and one for Abstract Meaning Representation (AMR) graphs—to improve medical dialogue generation. This allows the model to leverage the sequential nuances of natural conversation alongside the explicit logic of medical knowledge.
3. Nested and Interleaved (The "Sandwich")
A more integrated approach involves interleaving graph and sequence layers. Charles Sutton introduced the "Graph Sandwich" and the GREAT transformer, which interleave sequential processing with graph-based message-passing. This allows information to be distributed globally across the sequence before being refined through local structural constraints, which is particularly powerful for structured data like source code.
Preserving Context in the Graph
A recurring challenge in this field is that converting text into a graph often discards the very word order that makes language intelligible. To solve this, Liang Ding proposed the Recurrent Graph Syntax Encoder (RGSE), which treats graph nodes themselves as recurrent units (RNNCells). By doing so, the model captures syntactic dependencies and word order simultaneously, ensuring that the linear flow of the sequence is preserved even as it traverses the non-linear connections of a dependency tree.
Beyond Natural Language
The utility of these hybrid models extends into domains where data is naturally both sequential and structural:
- Chemistry: Kelong Mao developed the Graph Enhanced Transformer (GET) for retrosynthesis, fusing molecular graph topology with SMILES string sequences to ensure chemical validity.
- Visual Arts: Yonggang Qi introduced SketchLattice, which uses a GCN to encode spatial relationships in a sketch and an RNN to decode them into a vector sequence, bridging the gap between static images and the temporal process of drawing.
- Network Science: Bowen Jin developed PATTON for "text-rich networks," where nodes are documents and edges are semantic links, integrating GNN modules directly into Transformer layers to handle both the content of the papers and the structure of the citation graph.
By treating sequence and structure as complementary views of the same data, these hybrid architectures provide a more robust framework for reasoning, allowing AI to understand not just what is being said, but how the underlying concepts are truly connected.