Abstract
The theory of Conceptual Spaces is an influential cognitive-linguistic framework for representing the meaning of concepts. Conceptual spaces are constructed from a set of quality dimensions, which essentially correspond to primitive perceptual features (e.g. hue or size). These quality dimensions are usually learned from human judgements, which means that applications of conceptual spaces tend to be limited to narrow domains (e.g. modelling colour or taste). Encouraged by recent findings about the ability of Large Language Models (LLMs) to learn perceptually grounded representations, we explore the potential of such models for learning conceptual spaces. Our experiments show that LLMs can indeed be used for learning meaningful representations to some extent. However, we also find that fine-tuned models of the BERT family are able to match or even outperform the largest GPT-3 model, despite being 2 to 3 orders of magnitude smaller.
Introduction
In cognitive science, the theory of Conceptual Spaces suggests we understand the world through "quality dimensions"—primitive perceptual features such as hue, saturation, or mass. These spaces provide a valuable intermediate layer for explainable AI, bridging the gap between raw data and symbolic logic. Traditionally, constructing these spaces has relied on expensive human ratings, limiting their application to narrow domains. This research investigates whether Large Language Models (LLMs) can serve as a source for these grounded conceptual representations, potentially automating the creation of conceptual spaces.
Methodology
The study evaluated several model families across diverse conceptual dimensions:
- GPT-3 (Probabilistic Ranking): Models like
davinciwere evaluated using conditional probabilities for prompt completions (e.g., "It is known that [food item] tastes..."). - ChatGPT & GPT-4 (Direct Ranking): These models were prompted to rank lists of items directly due to API limitations on raw probabilities.
- Fine-tuned Baselines (DeBERTa): A DeBERTa-v3-large model was fine-tuned on commonsense datasets (McRae and CSLB) to compare specialized smaller models against general-purpose giants.
- Bi-encoders: A BERT-large bi-encoder was trained on a novel dataset of 109,000 concept-property pairs generated by ChatGPT.
The models were tested against four primary benchmarks: a professional sensory panel's Taste Dataset (590 items), a Mass Dataset (49 objects), and pairwise comparison datasets for Size and Height (500 pairs each).
Analysis: The "Davinci" Hallucination
A significant finding of this study is the presence of egregious ranking errors in large models, termed the "Davinci" hallucination. Despite its scale, GPT-3 (davinci) often fails to correctly rank basic perceptual properties. For instance, it may rank cooked green cabbage or Saint-agur cheese as "sweeter" than fruit cake or chocolate.
These errors suggest that LLMs may rely on surface-level textual associations rather than a deep, grounded understanding of taste. This is likely exacerbated by reporting bias, where common knowledge (e.g., "sugar is sweet") is rarely stated explicitly in training corpora, while more specific or unusual associations (e.g., "sweet wine") are overrepresented.
Results and Discussion
The experiments revealed a "size paradox" in the taste domain: the fine-tuned DeBERTa model (approx. 400M parameters) significantly outperformed GPT-3 davinci (175B parameters), achieving a Spearman correlation ($\rho$) of 69.1% compared to 55.0% for sweetness.
However, the hierarchy shifts when considering physical properties. While smaller models and GPT-3 struggle with mass and physical dimensions, GPT-4 demonstrates a remarkable leap in performance.
As shown in the results above, GPT-4 achieved near-perfect accuracy (99.1%) on height and size comparisons. Conversely, the fine-tuned DeBERTa model, despite its success in taste, produced a negative correlation (-8.9%) for mass, indicating that text-based fine-tuning does not inherently provide a "feel" for physical weight.
Conclusion
While LLMs show promise for learning conceptual spaces, their performance is highly domain-dependent. Smaller, fine-tuned models can outperform massive LLMs in specific sensory domains like taste, provided they are trained on high-quality data. In contrast, reasoning about the physical world (mass, size) appears to benefit significantly from the increased scale and reasoning capabilities of models like GPT-4. Future work should explore more complex, non-linear conceptual structures and the potential for using LLMs to generate high-quality synthetic training data for specialized smaller models.
Summary updated