While Large Language Models (LLMs) have demonstrated superior comprehension and expressiveness across a wide range of tasks, a critical question remains: do they truly "understand" the world, or are they merely mimicking statistical patterns? This empirical study reveals a significant blind spot in current models: they are notably better at understanding concrete objects (like "animals" or "furniture") than abstract ideas (like "fairness" or "logic"). By testing models on a newly constructed dataset, researchers demonstrate that even as models scale up, they consistently struggle to map the hierarchical relationships of abstract concepts.
The Challenge of Concept Understanding
In real-world applications, tasks require varying levels of abstraction. A reliable AI must be able to classify physical entities just as well as it reasons about human emotions or abstract logic. To investigate whether LLMs handle these two types of concepts equally well, this study focuses on hypernym discovery—a classic NLP task that tests the ability to identify if one word represents a broader category of another (e.g., recognizing that "furniture" is a hypernym of "bed").
Methodology and the D-Concept Dataset
To measure this comprehension, the authors introduced a new benchmark called D-Concept, derived from WordNet, a massive lexical database of semantic relations. WordNet's existing hierarchy explicitly divides nouns into an "abstract" branch and a "physical" (concrete) branch. The resulting dataset contains 10,000 samples, split into training, validation, and testing sets. Each sample is a pair of entities, where positive samples are actual hypernym/hyponym pairs at various distances in the WordNet tree, and negative samples are randomly paired.
The researchers evaluated six prominent pre-trained LLMs: BERT, T5, CLIP, OpenClip, GPT Embedding (text-similarity-ada-001), and ChatGPT.
Two different evaluation approaches were utilized:
- Linear Probing: For most models, the internal weights were "frozen," and a simple linear layer was added on top to perform binary classification (determining if a pair is a hypernym or not).
- Prompting: For ChatGPT, a zero-shot prompt was used ("Is {entity1} the hypernym of {entity2}?"), alongside definitions to help the model disambiguate the words.
Key Insights: The Abstraction Gap
Across every single model tested, performance on abstract concepts was significantly weaker than on concrete ones. LLMs appear to have a much harder time placing abstract ideas within a logical hierarchy.
Interestingly, scaling isn't a silver bullet. While larger models (like the GPT series) perform better overall than smaller ones (like BERT), the performance gap between abstract and concrete concepts persists. Scaling improves general accuracy but does not necessarily resolve the underlying difficulty with abstraction.
Furthermore, T-SNE visualizations of the models' embedding spaces reveal that models naturally cluster abstract and concrete concepts into two distinct groups.
This spatial distinction suggests that the models "know" these are fundamentally different types of concepts, yet they still struggle to map the internal logic and hierarchical relationships within the abstract cluster.
Quantitative Results
The quantitative results from the hypernym discovery task clearly highlight this disparity across all tested architectures.
Key takeaways from the results include:
- GPT Embedding Model: Achieved a strong 91.98% accuracy on concrete concepts, but this performance dropped to 89.45% on abstract ones.
- ChatGPT (Zero-shot): Exhibited the most dramatic gap, achieving 78.98% accuracy for concrete concepts compared to only 64.13% for abstract concepts. Its F1 score for abstract concepts was particularly low (44.10), indicating a severe struggle to identify these relationships without specific task training.
- Consistency Across Architectures: Vision-language models like CLIP and OpenClip followed the exact same trend, consistently performing 3–4 percentage points worse on abstract data.
Limitations and Future Work
While this study provides compelling evidence of an abstraction gap, it focuses solely on hypernym discovery. Though a classic NLP benchmark, it represents only one facet of true semantic "understanding."
Additionally, the evaluation methods varied slightly by necessity—ChatGPT was tested via zero-shot prompting, while other models were fine-tuned with a linear probe. While this makes a direct "apples-to-apples" comparison between ChatGPT and BERT difficult, the internal gap between abstract and concrete performance remains undeniably clear within each model's individual results.
Future research must investigate why this semantic gap exists in the embedding spaces of large language models and explore new training techniques designed to help LLMs better grasp high-level, non-physical concepts.
Summary updated