Deep Bidirectional Language-Knowledge Graph Pretraining

Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang, Jure Leskovec

Abstract

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks.

Introduction

Large Language Models (LLMs) like BERT and GPT are masters of linguistic patterns but often struggle with structured, multi-step reasoning or factual consistency. Conversely, Knowledge Graphs (KGs) provide a rigorous, structured "scaffold" of facts (e.g., Paris —[is_capital_of]—> France) but lack the fluid context of natural language. While previous research has tried to combine these modalities, most approaches are "shallow" or "unidirectional"—either adding entity features to an LM or using a KG to inform a model only during the final finetuning stage.

DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining) addresses this gap by training a foundation model to "think" using both natural language and structured knowledge graphs simultaneously. It is a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale.

Methodology

The DRAGON framework utilizes two core innovations to achieve deep bidirectional fusion:

  1. Cross-Modal Architecture (GreaseLM): Instead of simply concatenating text and graph data, DRAGON uses the GreaseLM architecture. This involves $N$ layers of a standard LM followed by $M$ "fusion layers." In these fusion layers, a Transformer (for text) and a Graph Neural Network (for the KG) exchange information through a special "interaction node," allowing the modalities to influence each other deeply at every step.
  2. Joint Self-Supervised Pretraining: DRAGON is trained on two tasks simultaneously:
  3. Masked Language Modeling (MLM): The model predicts missing words in a sentence, learning to use the structured facts in the KG as context.
  4. Link Prediction (LinkPred): The model predicts missing edges in the KG, learning to use the surrounding text to understand how entities relate.
Overview of the DRAGON pretraining pipeline, showing the bidirectional fusion of text and KG modalities through joint MLM and Link Prediction objectives.

Experimental Setup

The researchers evaluated DRAGON in two distinct environments:

  • General Domain: Pretrained on BookCorpus (text) and ConceptNet (KG). Evaluated on nine benchmarks, including CSQA, OpenbookQA, RiddleSense, and HellaSwag.
  • Biomedical Domain: Pretrained on PubMed abstracts (text) and the UMLS (Unified Medical Language System) knowledge graph. Evaluated on MedQA-USMLE, PubMedQA, and BioASQ.

Analysis & Discussion

The core finding is that bidirectionality is essential: pretraining on Link Prediction (a graph task) improves Language Modeling (a text task), and vice versa. The text provides the nuance that graphs lack, while the graph provides the logic that text often obscures.

A significant advantage of DRAGON is its performance on complex reasoning, particularly involving negation (e.g., "Which of these is not..."). Standard LMs often ignore "not," but DRAGON can use the KG to see which paths are explicitly absent or logically inconsistent.

Visualization of DRAGON's reasoning process on conjunction and negation tasks, demonstrating how the model uses KG structure to outperform text-only models.

Furthermore, DRAGON is remarkably data-efficient. In low-resource settings (using only 10% of available finetuning data), it significantly outperforms models that were only pretrained on text. The model’s performance also continues to improve as the number of fusion layers increases, suggesting strong scaling potential.

Key Results

  • General Domain: DRAGON outperformed RoBERTa (its base LM) by an average of +8% absolute accuracy across nine commonsense tasks.
  • Biomedical Domain: It set a new state-of-the-art on MedQA and PubMedQA, outperforming the previous best model (BioLinkBERT) by +3%.
  • Complex Reasoning: On questions requiring multi-step reasoning or involving long contexts, DRAGON achieved a +10% gain over existing models.
  • KG Link Prediction: Beyond NLP tasks, DRAGON also became a better graph reasoner, outperforming standard KG-only models (like DistMult) at predicting missing links.

Conclusion

DRAGON demonstrates the power of deep, bidirectional pretraining on text and knowledge graphs. While currently an encoder-only model—making it excellent for understanding but not for long-form generation—it sets a new standard for knowledge-augmented reasoning. Future work includes extending this architecture to encoder-decoder models for KG-enhanced generation and scaling the size of the retrieved KG subgraphs.

Summary updated

Abstract

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon.
BibTeX
@inproceedings{yasunaga2022deep,
  author = {Michihiro Yasunaga and Antoine Bosselut and Hongyu Ren and Xikun Zhang and Christopher D. Manning and Percy Liang and Jure Leskovec},
  title = {{Deep Bidirectional Language-Knowledge Graph Pretraining}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=4rFdbdeRjV},
}
Figure