Abstract
Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter, Iryna Gurevych, Hinrich Schütze. Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15).
2021.
Introduction
Generating natural language from structured Knowledge Graphs (KGs) is essential for making complex data accessible. However, existing methods often face a trade-off: Graph Neural Networks (GNNs) capture local structure well but struggle with long-range dependencies, while standard Transformers offer global reach but are "graph-blind," often ignoring the underlying topology. This is particularly problematic for real-world KGs, which are frequently disconnected or "messy."
To bridge this gap, we present Graformer, a parameter-efficient Transformer architecture that incorporates graph structure through relative position embeddings. By encoding shortest path lengths between nodes, Graformer allows the model to dynamically learn various structural views, balancing local connectivity with global context even in disconnected graphs.
Methodology
Graformer adapts the standard Transformer encoder to respect graph topology by redefining "position." In text, position refers to sequence distance; in Graformer, it refers to the shortest path length ($\delta$) between nodes.
Graph Representation
The model first transforms the KG into an incidence graph—a bipartite structure where both entities and relations are treated as nodes. To handle multi-word entities, these are further decomposed into a token graph. This unified representation allows the Transformer to attend to both entities and relations simultaneously.
Structural Multi-Head Attention
The core innovation is the use of Relative Position Embeddings. The attention mechanism is biased by the shortest path distance between any two nodes $i$ and $j$. The model distinguishes between forward paths, backward paths, and "unreachable" nodes ($\delta = \infty$). Because Graformer utilizes multi-head attention, different heads can specialize in different topological distances. For instance, one head might act as a local aggregator (focusing on immediate neighbors), while another captures global context by attending to distant or even unreachable nodes.
Experimental Setup
Graformer was evaluated on two key KG-to-text benchmarks:
- AGENDA: Scientific abstracts generated from automatically extracted (and often disconnected) entities and relations.
- WebNLG: Descriptions of DBPedia subgraphs, which are typically well-connected and human-authored.
Results and Discussion
Graformer demonstrates that topology can be effectively treated as an attention bias. Instead of rigid structural constraints, the model learns the graph's importance empirically during the generation task.
Learning Structural Views
Visualization of the attention heads confirms that the model learns to view the graph through different "lenses." In the AGENDA dataset, which features many disconnected components, the model notably dedicates specific attention heads to "unreachable" nodes. This suggests that understanding the absence of a connection is vital for maintaining global coherence in text generation.
Performance and Efficiency
Graformer achieves competitive performance while being significantly more parameter-efficient than state-of-the-art (SOTA) dual-encoder models:
- AGENDA: Achieved 98.8% of SOTA BLEU score using only 52% of the parameters (36.3M vs. 69.8M).
- WebNLG: Reached over 96% of SOTA performance with roughly half the parameters (5.3M vs. 10.4M).
- Comparison: It consistently outperformed "graph-blind" Transformers that only considered direct neighbors, validating the multi-hop relative position approach.
Conclusion
Graformer offers a lean, effective solution for KG-to-text generation by unifying local and global structural awareness within a single Transformer architecture. While it currently relies on path length, future work could incorporate edge labels into the relative position embeddings to further enhance semantic capture without sacrificing efficiency.
Summary updated