ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference

Amit Gajbhiye, Noura Al Moubayed, Steven Bradley

Abstract

ExBERT is a novel framework that enhances the BERT language model by dynamically retrieving and integrating structured facts from external Knowledge Graphs (KGs). By utilizing BERT itself to identify contextually relevant facts, ExBERT achieves state-of-the-art performance on Natural Language Inference (NLI) tasks. The framework demonstrates that a "smarter" model with access to a targeted library of facts can outperform significantly larger models that rely solely on internal memory.

Introduction

While Transformer-based models like BERT excel at capturing linguistic nuances, they often lack "grounding"—the ability to relate text to real-world facts and commonsense logic. In Natural Language Inference (NLI), a model must determine if a "Premise" entails a "Hypothesis." For instance, if a premise states, "A giant wave is approaching," and the hypothesis says, "The boys are about to be hit," a human understands the entailment because of the knowledge that waves crash and crashing objects hit things. BERT may fail such inferences if these specific relationships were not prominent in its training data. Previous attempts to integrate external knowledge often relied on rigid keyword matching, which introduced irrelevant "noise" that degraded model performance.

The ExBERT Framework

The authors introduce ExBERT (External Knowledge Enhanced BERT), which employs a three-stage pipeline to bridge the gap between raw text and structured knowledge.

Architecture of the ExBERT framework showing the Knowledge Integration layer and Mixture Model.

1. Contextual Retrieval (Selection & Ranking)

Rather than simple keyword matching, ExBERT uses BERT to generate embeddings for both the input text and potential facts from a Knowledge Graph (such as ConceptNet). Facts are ranked based on cosine similarity, ensuring that the retrieved knowledge is contextually relevant to the specific sentence pair.

2. Knowledge Integration Layer

As shown in the architecture diagram, the model utilizes Multi-head Attention to determine which retrieved facts are most critical for the specific inference task. This layer aligns the external knowledge with the premise-hypothesis representations.

3. The Mixture Model

The "secret sauce" of ExBERT is the Mixture Model, which uses learnable parameter matrices to adaptively balance the original text context with the newly integrated external knowledge. This mechanism prevents external facts from overwhelming the original semantic meaning of the input sentences.

Crucially, ExBERT uses the same BERT encoder for both the input text and the external facts. This design choice ensures that all representations exist in the same "mathematical language," improving efficiency and integration quality.

Experimental Setup

The researchers evaluated ExBERT on two major NLI benchmarks:

  • SNLI (Stanford Natural Language Inference): A large-scale dataset (570,000 pairs) focused on general domain commonsense.
  • SciTail: A challenging science-domain dataset (27,000 pairs) derived from school science exams.

For external knowledge sources, the model utilized ConceptNet for general commonsense and the Aristo Tuple knowledge graph for specialized science-domain facts.

Key Results and Insights

ExBERT established new performance benchmarks across both datasets:

  • SciTail: Achieved 95.9% accuracy, a significant improvement over the previous state-of-the-art.
  • SNLI: Achieved 91.5% accuracy, outperforming standard BERT-Base and even some BERT-Large configurations.

Analysis of Knowledge Integration

  • Contextual Retrieval is Essential: Using BERT to rank KG triples using contextual embeddings (bigrams and trigrams) effectively filters out irrelevant facts that would otherwise introduce noise.
  • Domain Specificity: On the SciTail dataset, performance was significantly higher when using the science-specific Aristo Tuple KG compared to the general-purpose ConceptNet.
  • The "Goldilocks" Zone: The study found that adding more facts is not always beneficial. For SNLI, accuracy peaked at 11 external facts before declining, suggesting that excessive external information can become a distraction.
  • Efficiency over Scale: ExBERT (built on BERT-Base with 110M parameters) outperformed BERT-Large (340M parameters). This indicates that providing a model with a "lookup table" of facts is more effective than simply increasing parameter count for memorization.

Conclusion and Future Work

ExBERT demonstrates that external knowledge integration is a powerful path toward more robust natural language reasoning. While the current two-step retrieval process adds some computational overhead, the gains in accuracy and efficiency over larger models are substantial. Future research directions include applying the framework to larger architectures like RoBERTa and exploring unstructured text snippets or complex graph-based embeddings as knowledge sources beyond simple triples.

Summary updated

BibTeX
@inproceedings{gajbhiye2021exbert,
  author = {Amit Gajbhiye and Noura Al Moubayed and Steven Bradley},
  title = {{ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference}},
  booktitle = {ICANN},
  year = {2021},
  doi = {10.1007/978-3-030-86383-8\_37},
}
Figure