Abstract
If a Large Language Model (LLM) answers "yes" to the question "Are mountains tall?" then does it know what a mountain is? Can you rely on it responding correctly or incorrectly to other questions about mountains? The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply a general understanding of concepts relevant to the anchor query. We propose conceptual consistency to measure a LLM's understanding of relevant concepts. This novel metric measures how well a model can be characterized by finding out how consistent its responses to queries about conceptually relevant background knowledge are. To compute it we extract background knowledge by traversing paths between concepts in a knowledge base and then try to predict the model's response to the anchor query from the background knowledge. We investigate the performance of current LLMs in a commonsense reasoning setting using the CSQA dataset and the ConceptNet knowledge base. While conceptual consistency, like other metrics, does increase with the scale of the LLM used, we find that popular models do not necessarily have high conceptual consistency. Our analysis also shows significant variation in conceptual consistency across different kinds of relations, concepts, and prompts. This serves as a step toward building models that humans can apply a theory of mind to, and thus interact with intuitively.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in answering complex questions, the nature of their "understanding" remains a subject of debate. Standard accuracy metrics are often content-agnostic; they confirm whether a model is correct but fail to reveal if the reasoning is grounded in a coherent internal world model. For instance, a model might correctly identify that a mountain peak reaches above the tree line without truly grasping the underlying concepts of "mountain" or "tree line." This raises concerns that LLMs may be performing high-level pattern recognition rather than building a stable conceptual framework.
To address this, we introduce Conceptual Consistency (CC), a novel metric designed to quantify a model's internal coherence. By measuring how well a model’s performance on a complex "anchor" task can be predicted by its grasp of relevant background concepts, we can better evaluate whether its successes are backed by genuine understanding.
Methodology
The framework for measuring Conceptual Consistency involves several key steps to link high-level reasoning with foundational knowledge.
Anchor Task and Background Knowledge
We utilize the CommonsenseQA (CSQA) dataset as our anchor task, which consists of multiple-choice questions requiring everyday reasoning. For each anchor question, we identify key concepts and traverse ConceptNet, a large-scale commonsense knowledge graph, to extract related facts. These facts cover 14 specific relations, such as AtLocation, CapableOf, and MadeOf.
Evaluation Framework
- Question Generation: Extracted facts are automatically translated into natural language questions. To prevent the model from succeeding through simple affirmative bias, we also generate "negative" background facts (false statements).
- Robust Prompting: To mitigate linguistic noise, we employ a majority-vote prompting procedure. This involves using multiple "meta-prompts" (e.g., "Is this true?" vs. "Answer yes or no:") and diverse answer pairs (e.g., "True/False" vs. "Right/Wrong") to identify the model's most stable belief.
- The CC Metric: Conceptual Consistency is calculated by determining the predictive power of a model's background knowledge scores over its success on the anchor question. Low consistency occurs when a model knows the background facts but fails the anchor task, or vice versa.
Results and Analysis
Our investigation across various model families, including OPT, GPT (EleutherAI), and T0, revealed several critical insights into the nature of LLM reasoning.
The Impact of Scale and Instruction Tuning
A primary finding is that conceptual consistency generally increases with model size. As models grow in parameters, they become significantly better at "connecting the dots" between foundational facts and complex reasoning tasks. However, even the largest models exhibit notable inconsistencies, suggesting that current architectures still struggle with perfect internal coherence.
Interestingly, we observed a trade-off with instruction tuning. The T0 model, despite being optimized for following instructions, showed lower conceptual consistency compared to its base counterparts. This suggests that while instruction tuning enhances zero-shot performance and helpfulness, it may inadvertently decouple a model's underlying conceptual knowledge from its task execution.
Relation Sensitivity and Bias
Consistency is not uniform across all types of knowledge. Models tend to be more consistent with physical causality (e.g., "causes" relations) than with social or internal motivations (e.g., "desires"). Furthermore, smaller models exhibited a strong "Yes-bias," highlighting the necessity of including negative background facts and robust prompting to accurately measure comprehension.
Conclusion and Future Work
Conceptual Consistency provides a vital lens for evaluating the internal coherence of LLMs. Our results suggest a hierarchy of skills where simple facts serve as the foundation for complex reasoning, a structure that becomes more stable as models scale.
However, limitations remain. Our current approach focuses on "1-hop" relations, whereas real-world reasoning often requires multi-step conceptual chaining. Future research should distinguish between conceptually related facts and those that are constructively required for a specific logical proof. Developing more stable, diverse datasets for comprehension testing remains a priority for building models that humans can interact with intuitively through a reliable theory of mind.
Summary updated