Unpacking Large Language Models with Conceptual Consistency

Pritish Sahu, Michael Cogswell, Yunye Gong, Ajay Divakaran

TL;DR

Can a language model genuinely understand the concepts it reasons about, or is it merely reciting surface patterns? This paper introduces conceptual consistency, a metric that tests whether a large language model’s (LLM) performance on an anchor question (e.g., CommonsenseQA) can be reliably predicted from its grasp of underlying background facts (mined from ConceptNet). The authors find that while conceptual consistency scales with model size, it remains surprisingly low overall, revealing erratic conceptual foundations beneath seemingly competent answers.

What Problem Does This Paper Address?

When a human interacts with another person, they naturally rely on a Theory of Mind—inferring internal beliefs and conceptual understanding to predict future behavior. If someone knows what a mountain is, you expect them to know that mountains are tall, made of rock, and located on land.

LLMs frequently get individual questions right, but it is unclear whether this reflects a structured conceptual understanding or isolated pattern matching. Existing consistency evaluations typically assess constructive consistency (e.g., semantic invariance to paraphrasing or sensitivity to negations). This work instead asks a broader epistemic question: Does an LLM’s knowledge of relevant background concepts predict whether it will answer a complex target query correctly?

Methods

The authors quantify conceptual consistency via a three-step framework:

  1. Background Knowledge Extraction & Negative Mining:
  2. For a given multiple-choice anchor query $(Q, S, A)$ from CommonsenseQA, relevant concepts are extracted and mapped to ConceptNet.
  3. Direct graph relations ($L=1$) between concepts are extracted as positive facts (e.g., (book, used for, school) $\rightarrow$ "Are books used for school?").
  4. Because smaller LLMs exhibit a severe affirmative ("yes") bias, the authors introduce a pairwise negative fact generation technique. For each positive tuple $(c^1, r, c^2)$, they uniformly sample an invalid concept $\bar{c}^2$ such that $(c^1, r, \bar{c}^2)$ is false, cyclic-free, and an English word.
  5. Robust Multi-Prompt Evaluation:
  6. Simple prompts suffer from lexical noise and grammatical artifacts. The authors evaluate 6 prompt templates crossed with 14 polar answer pairs (e.g., Yes/No, True/False, Correct/Incorrect)—evaluating $6 \times 14 = 84$ variations per query using single-token likelihoods to assign a robust positive/negative answer.
  7. Conceptual Consistency Metric ($CC$):
  8. A balanced background score $S_B^i \in [0, 1]$ measures model accuracy across positive and negative background facts for question $i$.
  9. Anchor score $S_A^i \in \{0, 1\}$ denotes correctness on the main task.
  10. Conceptual consistency is defined as the Average Precision (AP) when using background scores $S_B$ to predict task accuracy $S_A$: $$\text{CC} = \text{AP}(S_A, S_B)$$

Datasets

  • CommonsenseQA (CSQA): Used as the primary anchor task for zero-shot commonsense question answering (development set used, as test labels are private).
  • ConceptNet: Used as the commonsense background knowledge graph across 14 relations (e.g., AtLocation, CapableOf, Causes, Desires, MadeOf, UsedFor).
  • Evaluated Models: Decoder-only models from OPT (350M to 66B) and GPT-EleutherAI (125M to 6B), along with the multitask-prompted encoder-decoder T0 (3B and 11B).

Key Insights and Ideas

  • Consistency Scales with Model Size, but Plateaus: Larger models exhibit higher conceptual consistency (OPT climbs from $\sim$0.20 AP at 350M to $\sim$0.55 AP at 30B). However, OPT-30B and OPT-66B show nearly identical performance, hinting at diminishing returns.
  • T0 Underperforms Decoder-Only LLMs: Despite strong zero-shot instruction-following capabilities, T0 exhibits substantially worse conceptual consistency ($\sim$0.15–0.18 AP), potentially due to prompt sensitivity or fine-tuning side effects.
  • Skill Hierarchy (Bloom's Taxonomy): Scaling model size improves complex task performance (CSQA accuracy spans from $\sim$15% up to $\sim$55%) much more aggressively than background fact verification (which mostly fluctuates between 50% and 70%). Simple factual knowledge appears to saturate earlier than reasoning ability.
  • Relation-Specific Divergence: For some relations like "causes", all model scales show similar factual verification performance, but larger models use that knowledge far more consistently. Conversely, for "similar to", smaller models often out-consist larger ones.

Key Results

  • Conceptual Consistency ($CC$): In absolute terms, average precision across models remains modest, peaking at $\sim$0.55 for OPT-30B/66B and $\sim$0.45 for GPT-NeoX-6B, showing that model knowledge and anchor performance still frequently decouple.
  • Prompt & Affirmation Bias: Single-prompt baselines on small OPT models defaulted to saying "yes" over 90% of the time on background probes. Pairing negative facts and aggregating over 84 meta-prompt/token combinations substantially flattened this bias.

Limitations and Future Work

  • Heuristic Knowledge Mining: Restricting ConceptNet traversal to paths of length $L=1$ captures immediate relations, but misses higher-order multi-hop reasoning. In addition, automated templates can produce ungrammatical sentences (e.g., "Does mammals has a watch?").
  • Scope of Architecture & Scale: Evaluation was limited to open-weight models up to 66B parameters due to compute constraints, omitting closed contemporary frontier models.
  • Future Directions: Developing benchmark datasets explicitly organized around hierarchical skill levels and exploring whether human-understandable concept attribution can be used in real-time interactions to improve model predictability.

Summary updated

Abstract

If a Large Language Model (LLM) answers “yes” to the question “Are moun- tains tall?” then does it know what a mountain is? Can you rely on it respond- ing correctly or incorrectly to other questions about mountains? The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply a general understanding of concepts relevant to the anchor query. We propose conceptual consistency to measure a LLM’s understanding of relevant concepts. This novel metric measures how well a model can be characterized by finding out how con- sistent its responses to queries about conceptually relevant background knowledge are. To compute it we extract background knowledge by traversing paths between concepts in a knowledge base and then try to predict the model’s response to the anchor query from the background knowledge. We investigate the performance of current LLMs in a commonsense reasoning setting using the CSQA dataset and the ConceptNet knowledge base. While conceptual consistency, like other metrics, does increase with the scale of the LLM used, we find that popular mod- els do not necessarily have high conceptual consistency. Our analysis also shows significant variation in conceptual consistency across different kinds of relations, concepts, and prompts. This serves as a step toward building models that humans can apply a theory of mind to, and thus interact with intuitively.
BibTeX
@inproceedings{sahu2023unpacking,
  author = {Pritish Sahu and Michael Cogswell and Yunye Gong and Ajay Divakaran},
  title = {{Unpacking Large Language Models with Conceptual Consistency}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=YsAbPH2VWKE},
}

Loading related content...

Figure