Prompt-based methods may underestimate large language models' linguistic generalizations

Jennifer Hu, Roger Levy

Introduction

Evaluating what a Large Language Model (LLM) "knows" has become increasingly difficult as models move behind closed APIs. While early research focused on internal probability distributions (logits), contemporary evaluation often relies on prompting—asking the model to perform a task or judge a sentence in natural language. This paper argues that prompting introduces a "metalinguistic" hurdle that can obscure a model's true linguistic competence.

The Competence-Performance Gap

The authors distinguish between a model's internal linguistic knowledge (competence) and its ability to express that knowledge through a specific interface (performance). A model might internally assign a higher probability to a grammatical sentence than an ungrammatical one, yet fail to correctly identify the better sentence when asked via a complex prompt. This gap suggests that many reported "failures" of LLMs may be artifacts of the prompt interface rather than deficiencies in the model's underlying linguistic generalizations.

Methodology: Direct vs. Metalinguistic Evaluation

The researchers compared two primary evaluation strategies across six models (Flan-T5 and GPT-3/3.5 variants):

  1. Direct Method: This approach measures the raw output probabilities (logits) the model assigns to specific tokens or full sequences. It is considered the "ground truth" of the model's internal state.
  2. Metalinguistic Prompting: This involves zero-shot prompts that ask the model to perform linguistic tasks, such as predicting the next word or judging sentence grammaticality.
Comparison of Direct and Metalinguistic evaluation methods for word prediction.

The study tested these methods across four tasks: word prediction, semantic plausibility, and two types of syntactic judgment (isolated sentence judgment vs. minimal pair comparison). These tests utilized standard benchmarks like BLiMP and SyntaxGym, as well as a new dataset of recent news articles to ensure the models hadn't seen the text during training.

The Direct Method for sentence comparison involves calculating the total probability of two candidate sequences to determine which is more likely.

Key Results and Insights

The study revealed several critical findings regarding LLM evaluation:

  • Direct Methods Outperform Prompting: In nearly every task, direct probability measurements revealed more robust linguistic knowledge than prompting. For syntactic tasks, the direct method often achieved 80-90% accuracy, while prompting methods sometimes struggled to exceed a 50% random baseline.
  • The Alignment Penalty: Performance degrades as the prompt diverges from the model's core training objective (next-token prediction). Complex, verbose prompts often yielded the worst alignment with the model's internal knowledge.
  • Minimal Pairs are Essential: Models are much more successful when asked to compare two options (Sentence Comparison) rather than judging a single sentence in isolation. This suggests that "forced choice" evaluations are a more accurate probe of LLM knowledge.
  • Consistency Decay: While internal probabilities and prompted responses correlate well for simple tasks like word prediction, this correlation drops sharply for complex syntactic tasks, sometimes falling to near zero.

Conclusion and Implications

The findings highlight a significant risk in the current trend toward evaluating models solely through closed APIs that restrict access to log probabilities. If researchers cannot access internal logits, they may reach overly pessimistic conclusions about LLM capabilities. While larger models generally show better alignment between methods, the gap remains significant, suggesting that direct probability remains the gold standard for measuring true linguistic generalization.

Summary updated

BibTeX
@inproceedings{hu2023prompt,
  author = {Jennifer Hu and Roger Levy},
  title = {{Prompt-based methods may underestimate large language models' linguistic generalizations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.306},
}
Figure