ERNIE-NLI: Analyzing the Impact of Domain-Specific External Knowledge on Enhanced Representations for NLI

Lisa Bauer, Lingjia Deng, Mohit BansalDeeLIO @ NAACL 2021 · DOI

Abstract

Natural Language Inference (NLI) tasks often require external "world knowledge" that standard language models lack. This paper introduces ERNIE-NLI, a framework that incorporates domain-specific knowledge into the ERNIE architecture. By mapping fine-grained relations from external knowledge bases into positive and negative "NLI polarities," the authors demonstrate that targeted knowledge injection can significantly improve classification accuracy for specific NLI labels. Experimental results on the MNLI dataset show that positive knowledge boosts entailment detection, while balanced knowledge improves performance on neutral and contradiction labels.

Introduction

Natural Language Inference (NLI) is a fundamental task in NLP, requiring a model to determine if a hypothesis is entailed by, contradicts, or is neutral toward a premise. While large-scale pre-trained models like BERT have set new benchmarks, they often struggle with logic that requires external context. Error analysis on the Multi-Genre NLI (MNLI) dataset suggests that at least half of BERT's failures are due to a lack of world knowledge or domain-specific jargon. ERNIE-NLI seeks to bridge this gap by systematically integrating structured external knowledge into the model's representations.

The ERNIE-NLI Framework

The ERNIE-NLI pipeline consists of three distinct phases: Knowledge Extraction, Knowledge Training, and Integration. In the extraction phase, concept pairs are pulled from three major sources: ConceptNet (general knowledge), PPDB (paraphrases), and WordNet (lexical relations).

The training phase involves a novel "polarity" mapping, where relations are categorized as either positive (e.g., Synonym, IsA, Entails) or negative (e.g., Antonym, DistinctFrom). A BERT-based classifier is then trained to learn embeddings for these polarities, teaching the model the logical "flavor" of the connection between words. Finally, the ERNIE architecture's information fusion layer combines these knowledge embeddings with standard text tokens to produce enhanced representations.

The three-phase pipeline of ERNIE-NLI: Knowledge Extraction, Knowledge Training, and Integration.

Experimental Setup and Datasets

The primary benchmark for training and evaluation is the MNLI dataset, containing 433k sentence pairs. The external knowledge is sourced from ConceptNet, the Paraphrase Database (PPDB), and WordNet. The study evaluates several configurations, including "PC" (Positive-heavy knowledge) and "PCW&Bal" (balanced knowledge including WordNet), to determine how different knowledge distributions affect classification.

Results and Analysis

The core finding of this research is that knowledge polarity is the primary driver of performance gains. Injecting positive knowledge (synonyms) led to a 2.59% accuracy improvement in the entailment class over the baseline ERNIE model. Conversely, negative knowledge (antonyms) proved essential for identifying contradictions.

Performance comparison of ERNIE and ERNIE-NLI variants across NLI labels.

The authors also highlight the importance of balance; the "PCW&Bal" setup improved accuracy for Neutral (+1.47%) and Contradiction (+0.59%) labels. Without this balance, an over-reliance on positive relations can bias the model toward predicting entailment even in cases of neutrality or contradiction. Furthermore, ERNIE-NLI demonstrated robustness to "unseen" knowledge, as it could compute useful embeddings for word pairs not present in the training set, provided the relationship type was familiar.

Conclusion

ERNIE-NLI provides a robust framework for analyzing how external knowledge impacts NLI. The results underscore that the type and balance of knowledge injected are just as important as the quantity. While the model does not surpass massive state-of-the-art architectures like T5-11B in raw score, it offers a valuable analytical framework for understanding how structured knowledge can be effectively integrated into neural representations to resolve logical ambiguities.

Summary updated

BibTeX
@inproceedings{bauer2021ernie,
  author = {Lisa Bauer and Lingjia Deng and Mohit Bansal},
  title = {{ERNIE-NLI: Analyzing the Impact of Domain-Specific External Knowledge on Enhanced Representations for NLI}},
  booktitle = {DeeLIO @ NAACL},
  year = {2021},
  doi = {10.18653/v1/2021.deelio-1.7},
}
Figure