MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling

Parishad BehnamGhader, Hossein Zakerinia, Mahdieh Soleymani BaghshahNAACL 2021

TL;DR

This paper introduces MG-BERT, a novel approach to enhance BERT's Masked Language Modeling (MLM) capabilities by incorporating rich graph-based information. By augmenting BERT with a multi-graph that combines knowledge graphs and co-occurrence statistics, MG-BERT significantly improves performance on MLM tasks, demonstrating the power of integrating external knowledge into language models.

What Problem Does This Paper Address?

Pre-trained language models like BERT have achieved remarkable success in Natural Language Processing (NLP). However, they exhibit limitations in tasks requiring commonsense and pragmatic reasoning, particularly in the Masked Language Modeling (MLM) objective. Specifically, BERT can struggle to fill masked tokens accurately when the context alone is insufficient, sometimes producing contradictory predictions. This paper aims to address this by enriching BERT's understanding with external knowledge sources, going beyond just the immediate text context.

Methods

The core innovation of MG-BERT lies in its multi-graph augmentation of the BERT architecture. This involves:

  • Static Multi-Graph: This graph combines three types of information:
  • Knowledge Graph (KG): Leverages real-world facts and relationships between entities.
  • PMI Graph: Captures local co-occurrences of words within sliding windows, calculated using Pointwise Mutual Information (PMI).
  • TF-IDF Graph: Represents document-level co-occurrences of tokens. This static graph is processed by a Relational Graph Convolutional Network (R-GCN) layer, which enriches token embeddings with global information.

  • Dynamic Sentence Graph: A graph attention mechanism is employed over a dynamic graph representing the current input sentence. This allows the model to capture local, context-dependent relationships between tokens.

The graph module is integrated directly after BERT's initial token embedding layer. The outputs from the R-GCN and graph attention layers are then aggregated with BERT's positional and segment embeddings before being fed into the BERT encoder layers. The model is trained using the MLM objective, where it learns to predict masked tokens by considering both the textual context and the augmented graph information.

Datasets

  • Training: The WN18 knowledge graph was used as an unlabeled graph for training.
  • Evaluation: The model was evaluated on the following text datasets:
  • COLA (Corpus of Linguistic Acceptability)
    • SST-2 (Stanford Sentiment Treebank)
    • Brown Corpus

Key Insights and Ideas

  • Bridging the Commonsense Gap: The paper highlights that BERT's reliance solely on textual context can be a bottleneck for commonsense reasoning. Integrating external knowledge graphs and co-occurrence statistics effectively compensates for this limitation.
  • Synergy of Static and Dynamic Graphs: The combination of a static multi-graph (for global knowledge) and a dynamic sentence graph (for local context) proves crucial. The static graph provides consistent, rich embeddings, while the dynamic graph adapts to the specific nuances of each sentence.
  • Importance of Diverse Graph Information: The experiments show that different graph types (KG, PMI, TF-IDF) contribute uniquely to performance, suggesting that a comprehensive multi-graph approach is more effective than relying on a single source.
  • R-GCN for Relational Data: The use of R-GCN is a key enabler for effectively processing the multi-relational nature of the static graph, allowing for nuanced integration of different types of word relationships.

Key Results

The paper reports significant improvements in the MLM task when using MG-BERT compared to baseline BERT and other state-of-the-art models like ERNIE and VGCN-BERT.

  • Hits@k Metrics: On the COLA, SST-2, and Brown datasets, MG-BERT consistently achieved higher Hits@1 and Hits@5 scores, indicating a better ability to predict the correct masked tokens. For instance, on the COLA dataset, MG-BERT achieved 71.72% Hits@1, outperforming BERT (68.50%), ERNIE (69.57%), and VGCN-BERT (69.03%).
  • Perplexity: MG-BERT also demonstrated lower perplexity scores on the evaluated datasets, suggesting it predicts masked tokens with greater certainty and accuracy.
  • Ablation Studies: Experiments with variations of MG-BERT (e.g., MG-BERT(base) which omits the dynamic graph) confirmed the importance of both the static multi-graph and the dynamic sentence graph for achieving optimal performance.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of MG-BERT is influenced by dataset-specific hyperparameters (λκ, XP, λτ, Adyn). Fine-tuning these parameters is crucial for optimal results.
  • Computational Cost: Augmenting BERT with graph modules and attention mechanisms likely increases computational complexity and training time, although this is not explicitly quantified in the paper.
  • Future Directions: The authors suggest that further exploration of different graph structures, more sophisticated graph neural network architectures, and applying MG-BERT to other NLP tasks beyond MLM are promising avenues for future research. The potential to incorporate even richer external knowledge sources also remains an open area.

Summary updated

BibTeX
@inproceedings{behnamghader2021mg,
  author = {Parishad BehnamGhader and Hossein Zakerinia and Mahdieh Soleymani Baghshah},
  title = {{MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure