Deep Bidirectional Language-Knowledge Graph Pretraining

Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang, Jure Leskovec

TL;DR

DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining) is a novel self-supervised foundation model that deeply integrates text and knowledge graphs (KGs) from the ground up. By jointly training a cross-modal encoder to predict masked words in text and missing links in graphs, DRAGON achieves state-of-the-art performance on complex reasoning and low-resource question-answering tasks across both general and biomedical domains.

What Problem Does This Paper Address?

Language Models (LMs) are incredibly powerful, but they often struggle with complex, multi-step logical reasoning because they lack structured background knowledge. Knowledge Graphs (KGs)—which map entities and their relationships (e.g., Paris -> in -> France)—provide an excellent scaffold for this kind of reasoning.

While previous research has tried to combine LMs and KGs, existing methods fall short in two ways:

  1. Shallow Pretraining: Most pretraining methods only fuse text and KGs in a shallow or uni-directional way (e.g., just appending entity embeddings to text tokens).
  2. Finetuning-Only: Models that do feature deep, bidirectional fusion between text and KGs (like GreaseLM or QA-GNN) are typically only trained during the finetuning phase on specific, labeled downstream tasks.

There has been a lack of a true foundation model that learns deeply interactive, bidirectional representations of text and KGs at scale during the self-supervised pretraining phase.

Methods

To solve this, the authors introduce DRAGON. The method takes pairs of text segments and relevant KG subgraphs (retrieved via entity linking) and processes them through a deep cross-modal encoder.

DRAGON Architecture and Pretraining
  • Architecture: DRAGON utilizes the GreaseLM architecture, which uses Transformer layers to process text tokens and Graph Neural Networks (GNNs) to process KG nodes. Special "interaction tokens/nodes" allow the text and graph layers to exchange information bidirectionally at multiple depths in the network.
  • Novel Pretraining Objective: The core innovation is how DRAGON is pretrained. It unifies two self-supervised tasks:
  • Masked Language Modeling (MLM): Masking tokens in the text and asking the model to predict them.
  • Link Prediction (LinkPred): Dropping edges (relationships) in the KG and asking the model to predict them.

By optimizing these jointly, the text is forced to ground itself in the KG's structure, and the KG is forced to contextualize itself using the surrounding text.

Datasets

The authors pretrained and evaluated DRAGON in two distinct domains to prove its versatility:

  • General Domain Pretraining: BookCorpus (text) + ConceptNet (KG).
  • Biomedical Domain Pretraining: PubMed abstracts (text) + UMLS (KG).
  • Downstream Evaluation (General): 9 commonsense reasoning benchmarks including CSQA, OpenBookQA (OBQA), RiddleSense, ARC, CosmosQA, HellaSwag, PIQA, SIQA, and aNLI.
  • Downstream Evaluation (Biomedical): MedQA-USMLE, PubMedQA, and BioASQ.

Key Insights and Ideas

  • Joint pretraining is greater than the sum of its parts: Ablation studies show that using MLM and Link Prediction together performs significantly better than using either alone. The text helps deduce missing graph links, while the graph structure helps fill in missing text.
  • A scaffold for complex reasoning: DRAGON shines brightest on questions involving negation, conjunctions, and long contexts. The KG acts as a structural scratchpad, allowing the model to robustly track logic that trips up standard LMs.
Complex Reasoning Examples
  • Massive gains in data efficiency: Because DRAGON learns general-purpose reasoning during pretraining, it requires far less labeled data during finetuning. It excels in low-resource environments where standard models overfit or fail to generalize.
  • Scalability: Previous finetuning-only models (like GreaseLM) did not improve when researchers added more cross-modal fusion layers. DRAGON, however, does improve with increased capacity, suggesting that pretraining unlocks the ability to scale these hybrid models much further.

Key Results

  • General Domain: DRAGON outperforms the baseline LM (RoBERTa) and existing KG-augmented models (QAGNN, GreaseLM) across all 9 downstream tasks, with an average absolute gain of +5%.
  • Low-Resource & Complex Tasks: Achieved a massive +8% gain on OBQA and RiddleSense, and a +10% gain on questions specifically requiring multi-step reasoning, negation, or long-context comprehension.
  • Biomedical Domain: Set a new state-of-the-art on biomedical NLP tasks, beating the previous best LM (BioLinkBERT) by up to +3% accuracy on MedQA and PubMedQA.

Limitations and Future Work

  • Encoder-Only Architecture: DRAGON is currently an encoder-only model (analogous to BERT or RoBERTa). It is designed for understanding, classification, and multiple-choice QA, but it cannot perform open-ended language generation.
  • Future Work: The authors note that an important next step is to extend this deep bidirectional KG-text pretraining approach to generative models (like GPT or T5) to advance KG-enhanced text generation.

Summary updated

Abstract

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon.
BibTeX
@inproceedings{yasunaga2022deep,
  author = {Michihiro Yasunaga and Antoine Bosselut and Hongyu Ren and Xikun Zhang and Christopher D. Manning and Percy Liang and Jure Leskovec},
  title = {{Deep Bidirectional Language-Knowledge Graph Pretraining}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=4rFdbdeRjV},
}
Figure