How Reliable are Model Diagnostics?

Vamsi Aribandi, Yi Tay, Donald Metzler

TL;DR

This paper investigates the reliability of common model diagnostic tools for large language models, finding that popular likelihood-based and representation-based diagnostics can produce wildly different results even with minor changes to training setups. This instability suggests that current diagnostic methods may not be as robust as practitioners assume, potentially leading to misleading conclusions about model behavior.

What Problem Does This Paper Address?

As deep learning models, particularly large language models (LLMs), become more powerful and are deployed in critical domains like medicine and justice, understanding their inner workings and potential failure modes is paramount. Traditional metrics like accuracy are insufficient for this. Consequently, there's a growing interest in developing "diagnostic probes" – specialized tests designed to reveal specific learned behaviors, such as syntactic knowledge or biases. However, the reliability and robustness of these diagnostic tools themselves are not well-understood. This paper addresses the crucial question: how trustworthy are these diagnostic probes when we try to understand LLM behavior?

Methods

The authors focus on evaluating three specific diagnostic tasks: StereoSet and CrowS-Pairs (likelihood-based diagnostics) and SEATS (vector-space diagnostics). To assess reliability, they employ a rigorous experimental setup:

  • Multiple Pre-training Runs: They pre-train five BERT BASE and LARGE models using identical configurations but varying the random seed. This approach intentionally leads to different local minima in the loss landscape, simulating real-world scenarios where retraining might yield slightly different models.
  • Focus on Pre-trained Models: To isolate the impact of diagnostics, they primarily use pre-trained BERT models without fine-tuning, as prior work suggests pre-training patterns are often resilient.
  • Varied Diagnostic Approaches: For SEATS, they examine both sentence-level ([CLS]) representations and contextualized word representations, as well as clustering of these representations.

Datasets

  • StereoSet: A benchmark for measuring stereotypical bias through intra-sentence and inter-sentence examples.
  • CrowS-Pairs: Another dataset for measuring social biases, focusing on intra-sentence examples where likelihoods are conditioned differently than StereoSet.
  • SEATS (Sentence Encoder Association Tests): Extends Word Embedding Association Tests (WEATS) to sentence embeddings, measuring associations between concepts.
  • Downstream Tasks for Evaluation: SST-2, RTE, and QNLI from the GLUE benchmark were used to evaluate model performance after fine-tuning.

Key Insights and Ideas

  • Diagnostic Instability is Pervasive: The core finding is that both likelihood-based and representation-based diagnostics are surprisingly unstable. Minor changes, like a different random seed or choice of representation, can lead to significantly different conclusions about a model's behavior.
  • Local Minima Matter for Diagnostics: The study highlights that different local minima reached during pre-training can exhibit drastically different diagnostic profiles, even if their performance on downstream tasks is nearly identical. This suggests that diagnostics might be more sensitive to the specific optimization path than to fundamental model capabilities.
  • Likelihood-Based Diagnostics Can Be Erratic: For StereoSet and CrowS-Pairs, while overall scores might be stable, individual categories often show high variance. Some examples are even predicted with opposite labels across different runs, indicating model uncertainty that isn't captured by simple likelihood comparisons.
  • Representation-Based Diagnostics Lack Robustness: SEAT results frequently hover around the "neutral" mark and show erratic statistical significance. This implies that the spatial relationships between representations, which these diagnostics aim to capture, are not consistently structured across different training runs or even different representation types (sentence vs. word).
  • Downstream Performance Masks Diagnostic Differences: Perhaps most surprisingly, models that show wildly divergent diagnostic results often perform equivalently on standard downstream tasks. This raises concerns that diagnostics might not accurately reflect a model's practical utility or failure modes in real-world applications.

Key Results

  • High Variance in Likelihood-Based Diagnostics: Table 1 shows that for StereoSet, categories often have standard deviations over 2.5 percentage points. Figure 1 illustrates that many examples in CrowS-Pairs are assigned different labels across runs, with up to 3 models assigning one label and 2 the opposite.
  • Conflicting Conclusions from Diagnostics: The paper notes that for religious stereotypes ("Rel."), CrowS-Pairs detected strong bias while StereoSet detected almost none.
  • SEATs Show Erratic Significance and Effect Sizes: Figure 3 demonstrates that SEAT results can be on either side of the neutral mark (0) and have varying statistical significance, even for models with identical configurations but different seeds.
  • Equivalent Downstream Performance: Table 2 shows that fine-tuned checkpoints achieved largely equivalent performance on SST-2, RTE, and QNLI, despite their differing diagnostic behaviors. For example, BERT BASE models achieved 91.2±0.3% on SST-2, and BERT LARGE achieved 93.0±0.3%.

Limitations and Future Work

  • Limited Scope of Diagnostics: The authors acknowledge that their study is restricted to three specific diagnostics (two likelihood-based, one representation-based) and that future work is needed to assess the reliability of other diagnostic probes.
  • Focus on BERT: The experiments were conducted on BERT models. While BERT is a foundational LLM, its behavior might differ from newer architectures.
  • Future Work on Diagnostic Behavior: The authors express excitement about analyzing the correlation between model local minima in the loss landscape and diagnostic behavior. This could lead to a better understanding of why diagnostics are unreliable and potentially inform the design of more robust diagnostic tools.
  • Future Work on Optimization: Understanding these correlations might also help inform optimization processes for future LLMs to achieve more desirable properties.
  • Broader Configuration Testing: The authors recommend that researchers propose probes that are tested across a wider range of model and probe configurations, not just publicly available checkpoints.

Summary updated

Abstract

In the pursuit of a deeper understanding of a model's behaviour, there is recent impetus for developing suites of probes aimed at diagnosing models beyond simple metrics like accuracy or BLEU. This paper takes a step back and asks an important and timely question: how reliable are these diagnostics in providing insight into models and training setups? We critically examine three recent diagnostic tests for pre-trained language models, and find that likelihood-based and representation-based model diagnostics are not yet as reliable as previously assumed. Based on our empirical findings, we also formulate recommendations for practitioners and researchers.
BibTeX
@inproceedings{aribandi2021how,
  author = {Vamsi Aribandi and Yi Tay and Donald Metzler},
  title = {{How Reliable are Model Diagnostics?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.155},
}
Figure