Introduction
In the quest to understand what Large Language Models (LLMs) actually "know," researchers often use a technique called property inference. The logic is simple: if you can train a small model to map a word embedding (like the vector for "crow") to a set of human-annotated features (like "is black," "can fly," "is a bird"), then the original embedding must encode that knowledge.
However, this paper argues that such results may be a "Clever Hans" effect—where a model appears to perform a complex task but is actually relying on unintended cues. The authors tackle a fundamental question: Does high prediction accuracy in these mapping experiments actually prove that semantic knowledge has "emerged" in the embeddings, or are we just seeing the limits of the mapping algorithms and the geometric structure of the data?
Methodology
The core of the study involves mapping BERT-base (layer 0) embeddings to three well-known "feature norms": McRae (sparse categorical), Buchanan (sparse categorical), and Binder (dense ratings).
The authors compare two standard mapping techniques:
- Partial Least Squares Regression (PLSR): A linear method maximizing covariance between inputs and outputs.
- Feed-Forward Neural Networks (FFNN): A non-linear approach using a single hidden layer.
Crucially, the authors introduce a series of control experiments to stress-test these methods:
- Upper Bound Analysis: Mapping a dataset to itself ($Y \to Y$) to determine the maximum possible performance given the data's structure.
- Shuffle Baselines: Predicting random features that maintain the same sparsity as the original data.
- Taxonomic Corruption: Replacing actual semantic categories (e.g., "bird") with random ones (e.g., "building") to see if performance drops.
Key Results and Discussion
The Equivalence of Methods
One significant finding is that when properly tuned, linear PLSR and non-linear FFNNs perform almost identically. Previous claims that non-linear models are superior were likely due to suboptimal hyperparameter choices leading to overfitting. This suggests that the complexity of the mapping model doesn't necessarily unlock more "knowledge" from the embedding.
The Low Ceiling of Sparse Data
For sparse datasets like McRae, the "upper bound" of what the algorithm can even represent is very low. As shown in the results table, the system's actual performance is strikingly close to this ceiling.
In McRae, the system achieved an F1 of 0.25, while the theoretical upper bound was only 0.27. This suggests the results are almost entirely determined by the data's sparsity rather than the embedding's quality. If the "best possible" score is 0.27, a score of 0.25 doesn't necessarily indicate deep semantic understanding.
Geometry vs. Semantics
The authors demonstrate that these mappings primarily capture geometric similarity. If "dog" and "cat" are close in vector space, the mapping will assign them similar features. This looks like "knowledge," but it's actually just a reflection of the fact that similar words cluster together.
Further evidence comes from the "Nonsense Success" on the Binder dataset. Predicting a completely non-semantic feature—the difference in character length between words—yielded a correlation of 0.71, nearly identical to the 0.74 achieved using real semantic data. Additionally, replacing essential taxonomic features (e.g., changing "raven is a bird" to "raven is a building") resulted in almost no performance drop (F1 went from 0.25 to 0.23).
Conclusion
The study concludes that high performance in property inference tasks is often an artifact of the mapping process or the underlying geometry of the embedding space, rather than proof of semantic knowledge. Until evaluation measures can distinguish between simple geometric proximity and specific property possession, we should remain skeptical of claims that embeddings "contain" human-like concepts. Future work must focus on developing metrics that go beyond simple correlation to truly probe the semantic depth of LLMs.
Summary updated