Introduction
Evaluating what a Large Language Model (LLM) "knows" has become increasingly difficult as models move behind closed APIs. While early research focused on internal probability distributions (logits), contemporary evaluation often relies on prompting—asking the model to perform a task or judge a sentence in natural language. This paper argues that prompting introduces a "metalinguistic" hurdle that can obscure a model's true linguistic competence.
The Competence-Performance Gap
The authors distinguish between a model's internal linguistic knowledge (competence) and its ability to express that knowledge through a specific interface (performance). A model might internally assign a higher probability to a grammatical sentence than an ungrammatical one, yet fail to correctly identify the better sentence when asked via a complex prompt. This gap suggests that many reported "failures" of LLMs may be artifacts of the prompt interface rather than deficiencies in the model's underlying linguistic generalizations.
Methodology: Direct vs. Metalinguistic Evaluation
The researchers compared two primary evaluation strategies across six models (Flan-T5 and GPT-3/3.5 variants):
- Direct Method: This approach measures the raw output probabilities (logits) the model assigns to specific tokens or full sequences. It is considered the "ground truth" of the model's internal state.
- Metalinguistic Prompting: This involves zero-shot prompts that ask the model to perform linguistic tasks, such as predicting the next word or judging sentence grammaticality.
The study tested these methods across four tasks: word prediction, semantic plausibility, and two types of syntactic judgment (isolated sentence judgment vs. minimal pair comparison). These tests utilized standard benchmarks like BLiMP and SyntaxGym, as well as a new dataset of recent news articles to ensure the models hadn't seen the text during training.
Key Results and Insights
The study revealed several critical findings regarding LLM evaluation:
- Direct Methods Outperform Prompting: In nearly every task, direct probability measurements revealed more robust linguistic knowledge than prompting. For syntactic tasks, the direct method often achieved 80-90% accuracy, while prompting methods sometimes struggled to exceed a 50% random baseline.
- The Alignment Penalty: Performance degrades as the prompt diverges from the model's core training objective (next-token prediction). Complex, verbose prompts often yielded the worst alignment with the model's internal knowledge.
- Minimal Pairs are Essential: Models are much more successful when asked to compare two options (Sentence Comparison) rather than judging a single sentence in isolation. This suggests that "forced choice" evaluations are a more accurate probe of LLM knowledge.
- Consistency Decay: While internal probabilities and prompted responses correlate well for simple tasks like word prediction, this correlation drops sharply for complex syntactic tasks, sometimes falling to near zero.
Conclusion and Implications
The findings highlight a significant risk in the current trend toward evaluating models solely through closed APIs that restrict access to log probabilities. If researchers cannot access internal logits, they may reach overly pessimistic conclusions about LLM capabilities. While larger models generally show better alignment between methods, the gap remains significant, suggesting that direct probability remains the gold standard for measuring true linguistic generalization.
Summary updated