Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs

Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter, Iryna Gurevych, Hinrich Schütze2021 · DOI

Abstract

Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter, Iryna Gurevych, Hinrich Schütze. Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15).

2021.

Introduction

Generating natural language from structured Knowledge Graphs (KGs) is essential for making complex data accessible. However, existing methods often face a trade-off: Graph Neural Networks (GNNs) capture local structure well but struggle with long-range dependencies, while standard Transformers offer global reach but are "graph-blind," often ignoring the underlying topology. This is particularly problematic for real-world KGs, which are frequently disconnected or "messy."

To bridge this gap, we present Graformer, a parameter-efficient Transformer architecture that incorporates graph structure through relative position embeddings. By encoding shortest path lengths between nodes, Graformer allows the model to dynamically learn various structural views, balancing local connectivity with global context even in disconnected graphs.

Methodology

Graformer adapts the standard Transformer encoder to respect graph topology by redefining "position." In text, position refers to sequence distance; in Graformer, it refers to the shortest path length ($\delta$) between nodes.

Graph Representation

The model first transforms the KG into an incidence graph—a bipartite structure where both entities and relations are treated as nodes. To handle multi-word entities, these are further decomposed into a token graph. This unified representation allows the Transformer to attend to both entities and relations simultaneously.

Incidence graph representation with token-level nodes.

Structural Multi-Head Attention

The core innovation is the use of Relative Position Embeddings. The attention mechanism is biased by the shortest path distance between any two nodes $i$ and $j$. The model distinguishes between forward paths, backward paths, and "unreachable" nodes ($\delta = \infty$). Because Graformer utilizes multi-head attention, different heads can specialize in different topological distances. For instance, one head might act as a local aggregator (focusing on immediate neighbors), while another captures global context by attending to distant or even unreachable nodes.

Experimental Setup

Graformer was evaluated on two key KG-to-text benchmarks:

  1. AGENDA: Scientific abstracts generated from automatically extracted (and often disconnected) entities and relations.
  2. WebNLG: Descriptions of DBPedia subgraphs, which are typically well-connected and human-authored.

Results and Discussion

Graformer demonstrates that topology can be effectively treated as an attention bias. Instead of rigid structural constraints, the model learns the graph's importance empirically during the generation task.

Learning Structural Views

Visualization of the attention heads confirms that the model learns to view the graph through different "lenses." In the AGENDA dataset, which features many disconnected components, the model notably dedicates specific attention heads to "unreachable" nodes. This suggests that understanding the absence of a connection is vital for maintaining global coherence in text generation.

Attention head specialization across different path distances in the AGENDA dataset.

Performance and Efficiency

Graformer achieves competitive performance while being significantly more parameter-efficient than state-of-the-art (SOTA) dual-encoder models:

  • AGENDA: Achieved 98.8% of SOTA BLEU score using only 52% of the parameters (36.3M vs. 69.8M).
  • WebNLG: Reached over 96% of SOTA performance with roughly half the parameters (5.3M vs. 10.4M).
  • Comparison: It consistently outperformed "graph-blind" Transformers that only considered direct neighbors, validating the multi-hop relative position approach.

Conclusion

Graformer offers a lean, effective solution for KG-to-text generation by unifying local and global structural awareness within a single Transformer architecture. While it currently relies on path length, future work could incorporate edge labels into the relative position embeddings to further enhance semantic capture without sacrificing efficiency.

Summary updated

BibTeX
@misc{schmitt2021modeling,
  author = {Martin Schmitt and Leonardo F. R. Ribeiro and Philipp Dufter and Iryna Gurevych and Hinrich Schütze},
  title = {{Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs}},
  year = {2021},
  doi = {10.18653/v1/2021.textgraphs-1.2},
}
Figure