LM-Pub-Quiz: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models

Max Ploner, Jacek Wiland, Sebastian Pohl, Alan AkbikNAACL 2025

TL;DR

This paper introduces LM-Pub-Quiz, a Python framework and leaderboard designed to comprehensively evaluate the relational knowledge embedded within language models (LMs). It addresses limitations of previous probing methods by offering a more unbiased assessment and enabling fine-grained analysis, making it easier for researchers to understand and compare LM capabilities.

What Problem Does This Paper Address?

Large language models (LLMs) have become central to many NLP tasks, and a key aspect of their capability is the factual knowledge they encode internally – often referred to as "parametric knowledge." Evaluating this knowledge is crucial for understanding LM performance, monitoring changes during training (especially continual learning), and ensuring reliable downstream applications. Previous methods for probing this knowledge, however, suffered from issues like skewed answer distributions and limitations in comparing models trained with different objectives (e.g., masked vs. causal LMs). This made it difficult to get an accurate and unbiased picture of an LM's relational knowledge.

Methods

The core of LM-Pub-Quiz is built around the BEAR (Bidirectional Encoder-decoder Attention-based Representation) probing mechanism. BEAR reformulates knowledge probing as a ranking task, using relational triples from a knowledge base (KB) to construct multiple-choice questions. The LM's "pseudo log-likelihood" is used to rank these options, and if the correct fact is ranked highest, it's considered correctly answered. This approach is designed to be compatible with both causal and masked LMs, and it avoids issues with skewed answer distributions by using a ranking mechanism rather than a simple prediction.

LM-Pub-Quiz provides a Python framework that offers:

  • Standalone Evaluation: Researchers can easily evaluate their LMs using the BEAR probe.
  • Hugging Face Integration: Seamless integration with the Hugging Face TRANSFORMERS library, allowing direct use with models from the hub and integration into training pipelines.
  • Fine-grained Analysis: Beyond an overall score, the framework allows for analysis across different knowledge domains and relation cardinalities (e.g., one-to-one vs. one-to-many relationships).
  • Leaderboard: A public leaderboard tracks the performance of various LMs, encouraging community participation.

Datasets

The primary dataset used for evaluation is BEAR, which consists of 60 relations derived from the Wikidata knowledge base. These relations represent factual knowledge in the form of triples (e.g., "has-capital," "born-in," "crosses-river"). The paper also mentions experiments using T-REx (part of the LAMA benchmark) for evaluating continual learning. For training experiments, datasets like arXiv abstracts, literary texts from blbooks, and Wikipedia text (wikitext-103-v1) were used for domain adaptation and continual pre-training.

Key Insights and Ideas

  • Unbiased Evaluation is Key: The paper strongly emphasizes the need for unbiased knowledge probing, highlighting how previous methods could be misleading due to skewed answer distributions. BEAR and LM-Pub-Quiz aim to rectify this.
  • Ranking vs. Prediction: Reformulating probing as a ranking task using pseudo log-likelihood is a crucial innovation, enabling fair comparison between different LM architectures and training objectives.
  • Fine-grained Analysis Matters: An overall score is useful, but understanding what kind of knowledge an LM possesses (e.g., geographical vs. artistic) or how it handles different relation types (cardinality) provides much deeper insights.
  • [MASK]-Predict vs. Multiple Choice for Continual Learning: The paper suggests that traditional [MASK]-predict methods might overestimate catastrophic forgetting in continual learning. The multiple-choice format of LM-Pub-Quiz seems to provide a more nuanced view, showing less severe degradation. This is because [MASK]-predict can be fooled by contextually plausible but factually incorrect tokens, whereas the multiple-choice format forces a choice among specific, relevant options.

Key Results

  • Domain Performance: Models generally perform best on geographical questions and worst on "arts" and "movies" domains. Training on Wikidata-derived data (like wikitext) tends to improve scores the most.
  • Domain Adaptation Benefits: Fine-tuning GPT-2 on arXiv abstracts significantly improved its performance on scientific knowledge.
  • Model Biases: The study identified biases in models, with roberta-base showing bias towards "South America" and "Antarctica," while GPT-2 and Mistral-7B-v0.1 leaned towards "Europe." The paper notes that while Mistral-7B-v0.1 showed strong biases in one metric, its overall accuracy suggested subject-specific knowledge could mitigate this.
  • Continual Learning Forgetting: When evaluating continual learning, LM-Pub-Quiz indicated a less severe performance degradation compared to [MASK]-predict methods, suggesting that traditional methods might be overly pessimistic about knowledge loss.

Limitations and Future Work

  • Limitations: The paper acknowledges that the BEAR probe, while comprehensive, is still a subset of Wikidata and may not cover all possible relational knowledge. The [MASK]-predict method's limitations in evaluating continual learning are highlighted as a key finding that motivates the use of LM-Pub-Quiz.
  • Future Work: The authors plan to extend the BEAR probe to additional knowledge bases to cover a wider range of knowledge domains. They also encourage community participation in extending the leaderboard and contributing to the library's development for further use cases.

Summary updated

BibTeX
@inproceedings{ploner2025lm,
  author = {Max Ploner and Jacek Wiland and Sebastian Pohl and Alan Akbik},
  title = {{LM-Pub-Quiz: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure