BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models

Jacek Wiland, Max Ploner, Alan AkbikNAACL 2024 · DOI

Sure, here's a blog-style summary of the paper "BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models":

TL;DR

This paper introduces BEAR, a novel framework for evaluating the factual knowledge of language models (LMs) that works for both masked and causal LMs. Unlike previous methods that are limited to specific LM types and answer formats, BEAR uses a log-likelihood ranking approach on a curated dataset, enabling a more comprehensive and unified evaluation of LM knowledge.

What Problem Does This Paper Address?

Pre-trained language models (LMs) have become foundational in NLP, storing vast amounts of factual and relational knowledge. Evaluating this knowledge is crucial for understanding LM capabilities and guiding development. However, existing methods, like the popular LAMA probe, have significant limitations:

  • LM Type Restriction: LAMA relies on the masked language modeling objective, making it unsuitable for causal LMs (e.g., GPT family). This prevents a unified comparison across different LM architectures.
  • Single-Subtoken Answers: LAMA can only test for single-token answers, discarding many factual relations with multi-token entities (e.g., "New York City").
  • Data Biases: Existing datasets often suffer from imbalanced answer distributions, overly revealing entity names, and issues with multiple correct answers, leading to inaccurate evaluations.

These limitations hinder the ability to accurately measure and compare the relational knowledge encoded in the ever-growing landscape of LMs.

Methods

The authors propose BEAR (Benchmark for Evaluating Associative Reasoning), a unified framework that addresses the aforementioned limitations. The core idea is to leverage an LM's inherent ability to estimate the log-likelihood of a given textual statement.

Here's how it works:

  1. Log-Likelihood Ranking: Instead of predicting a single masked token, BEAR presents a set of answer options for a given relation instance. For each option, a natural language statement is constructed using a template, the subject, and the answer. The LM then assigns a log-likelihood score to each statement.
  2. Unified Evaluation: The statements are ranked by their log-likelihood scores, and the model's performance is evaluated based on whether the correct answer is ranked highest. This approach works for both masked LMs (using a pseudo log-likelihood estimation) and causal LMs.
  3. Flexible Answer Space: BEAR imposes no restrictions on the length or format of answers, allowing for multi-token entities.
  4. Novel Dataset Construction: The authors created a new evaluation dataset, BEAR, by curating relations from Wikidata and addressing issues like answer skews, template biases, and rare Wikidata entries. They also ensured a balanced answer space and focused on knowledge likely found in general corpora.

Datasets

  • LAMA (T-REx subset): Used as a baseline and for comparison.
  • BEAR Dataset: A newly created dataset designed to overcome the limitations of previous benchmarks. It consists of 78 relations and 7,731 instances (with a larger variant containing 40,916 instances). Key features include balanced answer spaces, a focus on general knowledge, and no restrictions on answer length.

Key Insights and Ideas

  • Unified Probing is Possible: The log-likelihood ranking approach successfully bridges the gap between evaluating masked and causal LMs, enabling a truly unified framework.
  • Data Quality Matters Immensely: The careful curation of the BEAR dataset, addressing issues like answer skews and overly revealing entity names, significantly impacts LM performance and provides a more reliable evaluation.
  • Template Sensitivity: LMs can be highly sensitive to the phrasing of evaluation templates, highlighting the importance of robust template design and evaluation across multiple templates.
  • Larger Models, Better Knowledge: As expected, larger LMs consistently outperform smaller ones, demonstrating a clear trend of improved relational knowledge with increased model size.

Key Results

  • BEAR is Challenging: Scores across all models are generally low, indicating that BEAR is a demanding benchmark that effectively probes factual knowledge even with strong distractors.
  • Larger Models Excel: Llama-2-13b-hf achieved the highest score, with other large models like Mistral-7B-v0.1 and Gemma-7b also performing well.
  • Masked LMs Slightly Edge Out Causal LMs: Masked LMs showed a slight advantage over causal LMs, potentially due to the bidirectional nature of their pre-training objective being more effective for factual recall.
  • Multilingual Models Lag: XLM-ROBERTa models performed poorly, hypothesized to be due to their multilingual pre-training and BEAR's focus on English entities.
  • Template Impact: Performance can drop significantly based on template phrasing, with BERT models showing more sensitivity than some newer models.

Limitations and Future Work

  • Knowledge Scope: BEAR, like its predecessors, tests only factual, relational knowledge and does not assess broader commonsense reasoning or physical reasoning.
  • Wikipedia Bias: Despite efforts to mitigate it, there's a potential bias towards entities overrepresented on Wikipedia, which could favor LMs trained on Wikipedia.
  • Potential for Misuse: The authors acknowledge that knowledge probing tools could be misused by malicious actors.
  • Future Work: The authors suggest further research into the impact of pre-training data, exploring different pseudo log-likelihood metrics, and investigating the relationship between model confidence and prediction accuracy. They also emphasize the importance of summating token log-likelihoods for both masked and causal LMs in future experiments.

Summary updated

BibTeX
@inproceedings{wiland2024bear,
  author = {Jacek Wiland and Max Ploner and Alan Akbik},
  title = {{BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.155},
}

Loading related content...

Figure