Abstract
Large Language Models (LLMs) have achieved remarkable performance on standard question-answering (QA) benchmarks. However, it remains unclear whether these models possess a genuine understanding of underlying concepts or are simply world-class pattern matchers. This paper introduces a systematic framework to "probe" QA models using structured expert knowledge. By isolating fundamental building blocks of intelligence—definitions and taxonomic reasoning—we reveal that while state-of-the-art transformers like RoBERTa possess significant prior knowledge, their reasoning remains fragile when faced with multi-hop logic or adversarial distractors.
Introduction: The Problem of "Black Box" Knowledge
Existing QA benchmarks often conflate multiple reasoning skills, such as logic, retrieval, and common sense, into single questions. This complexity makes it difficult to determine why a model succeeds or fails. To address this, we propose a methodology to isolate and test two core components of semantic understanding: definitions (knowing what a word means) and taxonomic reasoning (understanding "ISA" relationships, such as "a poodle is a type of dog").
Methods: Controlled Probing Framework
The proposed framework utilizes expert knowledge sources to automatically generate thousands of multiple-choice questions. Unlike noisy crowdsourced data, these probes are derived from structured databases like WordNet and GCIDE.
A central technique in this study is Inoculation by Fine-Tuning. If a model can master a new, specific task (e.g., word sense disambiguation) with minimal examples (approx. 100) without degrading performance on its original training data, it suggests the model already possessed the underlying concept and merely needed to learn the specific output format. To increase difficulty, we implemented adversarial distractor generation, selecting "sister" concepts from the same taxonomic level to test fine-grained distinctions.
Probing Suites: WordNetQA and DictionaryQA
We introduced two major probe suites:
- WordNetQA: Derived from WordNet, featuring sub-tasks for Hypernymy (generalization), Hyponymy (specification), Synonymy, and Definitions.
- DictionaryQA: Derived from the GCIDE (GNU Collaborative International Dictionary of English), focusing on contextual word-sense disambiguation.
Models were initially trained on science-domain benchmarks, including OpenBookQA, ARC Challenge, and SciQ, before being subjected to these probes.
Key Insights: The Fragility of Reasoning
Our analysis yielded several critical findings regarding the nature of transformer-based knowledge:
- Structural Predisposition: Even in zero-shot settings, RoBERTa significantly outperformed task-specific LSTMs, suggesting that pre-training on massive corpora imparts a structural understanding of conceptual relationships.
- The "Hop" Penalty: Reasoning performance degrades predictably as the number of "hops" in a taxonomy increases. While a model may correctly identify that a "poodle is a dog" (1-hop), it often fails to consistently conclude that a "poodle is a mammal" (2-hops).
- Consistency vs. Accuracy: We introduced a "Strict Cluster" metric to evaluate whether a model can correctly answer every question related to a single concept. While instance-level accuracy was often high, cluster-level consistency was significantly lower, indicating that model knowledge is frequently "spotty" rather than holistic.
Results
- Zero-Shot Performance: RoBERTa achieved 77.1% accuracy on the Definitions probe without task-specific training, vastly outperforming non-transformer baselines.
- Rapid Inoculation: With just 3,000 examples, RoBERTa’s Hypernymy accuracy rose to 87.7%, with a negligible "inoculation cost" (less than 1% drop) on original science benchmarks.
- The Human Gap: Despite high scores, models still lag behind human performance. On gold-standard sets, humans scored 91-96%, while models struggled particularly with complex hyponymy (specific types).
Limitations and Future Work
While these probes offer more control than standard benchmarks, they remain synthetic, carrying a risk that models exploit "silver-standard" artifacts. Future research should extend this framework to more complex domains such as causal reasoning and temporal reasoning. The ultimate objective is to develop "certificates of understanding," requiring models to demonstrate consistent knowledge across entire semantic clusters.
Summary updated