Abstract
Conceptual spaces represent entities in terms of their primitive semantic features. Such representations are highly valuable but they are notoriously difficult to learn, especially when it comes to modelling perceptual and subjective features. Distilling conceptual spaces from Large Language Models (LLMs) has recently emerged as a promising strategy, but existing work has been limited to probing pre-trained LLMs using relatively simple zero-shot strategies. We focus in particular on the task of ranking entities according to a given conceptual space dimension. Unfortunately, we cannot directly fine-tune LLMs on this task, because ground truth rankings for conceptual space dimensions are rare. We therefore use more readily available features as training data and analyse whether the ranking capabilities of the resulting models transfer to perceptual and subjective features. We find that this is indeed the case, to some extent, but having at least some perceptual and subjective features in the training data seems essential for achieving the best results.
Introduction
While modern Knowledge Graphs (KGs) excel at storing discrete facts, they often struggle with graded properties—features that exist on a spectrum, such as the "healthiness" of a recipe or the "originality" of a film. These features are the building blocks of Conceptual Spaces, a framework where entities are defined by quality dimensions.
Learning these dimensions is historically difficult due to reporting bias: humans rarely document the obvious (e.g., that a banana is yellow), leading zero-shot LLMs to reflect text frequency rather than physical reality. Furthermore, the lack of massive human-ranked datasets for subjective dimensions like "sweetness" or "scary-ness" makes traditional fine-tuning difficult. This paper explores whether LLMs can learn to rank entities along these dimensions by transferring knowledge from more accessible numerical data.
Methodology
The study compares two primary strategies for extracting rankings from LLMs:
- Pointwise Approach: The model assigns a score to a single entity for a specific feature (e.g., "On a scale of 1-10, how sweet is a banana?").
- Pairwise Approach: The model compares two entities directly (e.g., "Is a banana sweeter than a lemon?").
To aggregate binary pairwise comparisons into a final sorted list, the authors tested a simple Count method and a more sophisticated SVM-based strategy. The latter uses a Linear Support Vector Machine to learn a weight vector that maximizes the margin between positive and negative pairwise examples, representing a novel application of SVMs for LLM ranking aggregation. Models including Llama 2 (7B/13B), Llama 3 (8B), and Mistral (7B) were fine-tuned using QLoRA for computational efficiency.
Experimental Setup
The researchers utilized a diverse mix of factual, perceptual, and subjective datasets for evaluation:
- Wikidata (WD): 20 rankings based on numerical attributes like river length and population.
- Taste: Ratings for 590 food items across dimensions like sweetness and saltiness.
- Rocks: 30 rock types ranked by physical properties like roughness.
- Tag Genome (TG): Subjective rankings for movies and books based on tags like "scary" or "funny."
- Physical Properties (Phys): Ground-truth data for mass, size, and height.
Results and Analysis
Quantitative results indicate that the Pairwise approach generally yields more accurate rankings than the Pointwise method across most domains. Llama 3-8B emerged as a top performer, often outperforming zero-shot GPT-4 in specific domains after being fine-tuned on relevant data.
A significant finding was the "Easy-to-Hard" Transfer: fine-tuning on factual data (like Wikidata population) improved the model's ability to rank subjective features (like movie genres). However, the best results in perceptual domains were only achieved when at least some subjective data was included in the training set.
Qualitative analysis confirms that the fine-tuned models capture intuitive human rankings. For example, the models correctly identified "caramel custard" as highly sweet while ranking "tripe" at the bottom of the scale. Similarly, it successfully distinguished between the horror of The Shining and the comedy of Kung Fu Panda.
Discussion and Conclusion
Despite the success of fine-tuning, models still exhibit occasional "hallucinatory" errors, such as ranking a "spaceship" as a suitable family vehicle due to frequent text associations. Furthermore, subjective rankings reflect the majority opinions present in training data and may not account for cultural nuances. Future work will focus on constructing higher-quality, comprehensive training sets for conceptual spaces to further bridge the gap between factual knowledge and human-like conceptual understanding.
Summary updated