Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

Qian Dong, Yiding Liu, Suqi Cheng, Shuaiqiang Wang, Zhicong Cheng, Shuzi Niu, Dawei Yin

Introduction

In modern information retrieval, passage re-ranking serves as a critical second stage, where a model re-orders a small set of candidate documents to ensure the most relevant results appear at the top. While Pre-trained Language Models (PLMs) such as BERT and ERNIE have significantly advanced the state-of-the-art in this field, they face two primary challenges: vocabulary mismatch and a lack of specialized domain knowledge.

Vocabulary mismatch occurs when a user's query and a relevant document use different terms for the same concept (e.g., "liver disease" vs. "hepatitis"). Furthermore, while PLMs possess vast "implicit" knowledge within their neural weights, they often lack the "explicit," structured factual knowledge required for specialized fields like medicine or law. To address these gaps, this research introduces the Knowledge Enhanced Re-ranking Model (KERM), a framework designed to inject explicit facts from knowledge graphs (KGs) into the re-ranking process.

Methodology: The KERM Framework

KERM operates through a two-phase process: Knowledge Graph Distillation and Knowledge Aggregation.

1. Knowledge Graph Distillation

Raw knowledge graphs, such as ConceptNet, are often too large and noisy for direct application. KERM refines this data in two steps:

  • Global Pruning: The model utilizes TransE embeddings to identify and remove unreliable or noisy triples from the KG.
  • Local Meta-Graph Construction: For each query-passage pair, the model identifies the most relevant sentence in the passage. It then extracts entities from both the query and that specific sentence, finding $k$-hop paths between them in the KG. This creates a "meta-graph" that acts as a semantic bridge, linking the user's intent directly to the document's content through established factual relationships.

2. Knowledge Aggregation

After constructing the meta-graph, KERM employs a dual-encoder architecture to fuse text and knowledge:

  • Text Encoder: A standard PLM (e.g., ERNIE) processes the raw text of the query and passage.
  • Knowledge Injector: This module consists of transformer layers integrated with a Graph Meta Network (GMN). The GMN propagates information across the meta-graph, allowing the model to reason about entity relationships.
  • Dynamic Fusion: A novel injector module fuses the text representations with the graph representations, ensuring that explicit facts actively inform the final relevance score calculation.

Experimental Evaluation

The researchers evaluated KERM across three diverse benchmarks to test its general and specialized performance:

  1. MSMARCO-DEV: A large-scale general domain dataset.
  2. TREC 2019 DL: A version of MSMARCO featuring fine-grained relevance labels.
  3. Ohsumed: A specialized bio-medical dataset used to evaluate performance in data-scarce professional domains.

Key Results

KERM demonstrated superior performance over strong baselines, including monoBERT and duoBERT. On the MSMARCO-DEV set, it achieved an MRR@10 of 40.1, matching or exceeding the performance of significantly larger models.

The model's impact was most pronounced in specialized domains. On the bio-medical Ohsumed dataset, KERM improved the MRR@10 from 30.5 to 32.1 compared to standard ERNIE, highlighting its ability to provide a "safety net" of explicit facts where implicit pre-trained knowledge may be insufficient.

Discussion and Insights

The Semantic Bridge

The effectiveness of KERM lies in the "paths" within the meta-graph. These paths provide a structural context that helps the model understand that disparate terms are semantically linked, effectively bridging the vocabulary gap that often hinders traditional PLMs.

Architectural Trade-offs

Experimental analysis revealed a "sweet spot" for knowledge injection. The authors found that using three injection layers provided the optimal balance; adding too many layers can introduce noise or cause the model to lose the general language patterns learned during pre-training. Additionally, by focusing on "key sentences" for graph construction rather than the entire passage, the authors maintained manageable inference times.

Conclusion and Future Work

KERM successfully demonstrates that injecting explicit knowledge into PLMs can significantly enhance passage re-ranking, particularly in technical or specialized niches. However, several challenges remain for future research:

  • Computational Efficiency: Building meta-graphs "on the fly" for query-passage pairs is computationally intensive for massive production environments.
  • End-to-End Learning: Moving from heuristic-based pruning to a jointly learned graph generator could further improve model flexibility.
  • Retrieval Scaling: Extending these explicit knowledge techniques from the re-ranking stage to the initial retrieval stage (searching millions of documents) remains a significant open challenge in the field.

Summary updated

BibTeX
@inproceedings{dong2022incorporating,
  author = {Qian Dong and Yiding Liu and Suqi Cheng and Shuaiqiang Wang and Zhicong Cheng and Shuzi Niu and Dawei Yin},
  title = {{Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2022},
  url = {https://openreview.net/forum?id=Mbj1ms61w_},
}
Figure