Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings

Hanna Herasimchyk, Alhassan Abdelhalim, Sören Laue, Michaela RegneriarXiv 2025 · DOI · arXiv

Introduction

In the quest to understand what Large Language Models (LLMs) actually "know," researchers often use a technique called property inference. The logic is simple: if you can train a small model to map a word embedding (like the vector for "crow") to a set of human-annotated features (like "is black," "can fly," "is a bird"), then the original embedding must encode that knowledge.

However, this paper argues that such results may be a "Clever Hans" effect—where a model appears to perform a complex task but is actually relying on unintended cues. The authors tackle a fundamental question: Does high prediction accuracy in these mapping experiments actually prove that semantic knowledge has "emerged" in the embeddings, or are we just seeing the limits of the mapping algorithms and the geometric structure of the data?

Methodology

The core of the study involves mapping BERT-base (layer 0) embeddings to three well-known "feature norms": McRae (sparse categorical), Buchanan (sparse categorical), and Binder (dense ratings).

The standard mapping experiment setup: Encodings (source matrix) are mapped to Feature Norms (target matrix) during training, then evaluated on unseen concepts.

The authors compare two standard mapping techniques:

  1. Partial Least Squares Regression (PLSR): A linear method maximizing covariance between inputs and outputs.
  2. Feed-Forward Neural Networks (FFNN): A non-linear approach using a single hidden layer.

Crucially, the authors introduce a series of control experiments to stress-test these methods:

  • Upper Bound Analysis: Mapping a dataset to itself ($Y \to Y$) to determine the maximum possible performance given the data's structure.
  • Shuffle Baselines: Predicting random features that maintain the same sparsity as the original data.
  • Taxonomic Corruption: Replacing actual semantic categories (e.g., "bird") with random ones (e.g., "building") to see if performance drops.

Key Results and Discussion

The Equivalence of Methods

One significant finding is that when properly tuned, linear PLSR and non-linear FFNNs perform almost identically. Previous claims that non-linear models are superior were likely due to suboptimal hyperparameter choices leading to overfitting. This suggests that the complexity of the mapping model doesn't necessarily unlock more "knowledge" from the embedding.

The Low Ceiling of Sparse Data

For sparse datasets like McRae, the "upper bound" of what the algorithm can even represent is very low. As shown in the results table, the system's actual performance is strikingly close to this ceiling.

Performance comparison across different norms and control conditions. Note how close the system performance (Sys) is to the theoretical Upper Bound for sparse datasets like McRae.

In McRae, the system achieved an F1 of 0.25, while the theoretical upper bound was only 0.27. This suggests the results are almost entirely determined by the data's sparsity rather than the embedding's quality. If the "best possible" score is 0.27, a score of 0.25 doesn't necessarily indicate deep semantic understanding.

Geometry vs. Semantics

The authors demonstrate that these mappings primarily capture geometric similarity. If "dog" and "cat" are close in vector space, the mapping will assign them similar features. This looks like "knowledge," but it's actually just a reflection of the fact that similar words cluster together.

Further evidence comes from the "Nonsense Success" on the Binder dataset. Predicting a completely non-semantic feature—the difference in character length between words—yielded a correlation of 0.71, nearly identical to the 0.74 achieved using real semantic data. Additionally, replacing essential taxonomic features (e.g., changing "raven is a bird" to "raven is a building") resulted in almost no performance drop (F1 went from 0.25 to 0.23).

Conclusion

The study concludes that high performance in property inference tasks is often an artifact of the mapping process or the underlying geometry of the embedding space, rather than proof of semantic knowledge. Until evaluation measures can distinguish between simple geometric proximity and specific property possession, we should remain skeptical of claims that embeddings "contain" human-like concepts. Future work must focus on developing metrics that go beyond simple correlation to truly probe the semantic depth of LLMs.

Summary updated

Abstract

Understanding what knowledge is implicitly encoded in deep learning models is essential for improving the interpretability of AI systems. This paper examines common methods to explain the knowledge encoded in word embeddings, which are core elements of large language models (LLMs). These methods typically involve mapping embeddings onto collections of human-interpretable semantic features, known as feature norms. Prior work assumes that accurately predicting these semantic features from the word embeddings implies that the embeddings contain the corresponding knowledge. We challenge this assumption by demonstrating that prediction accuracy alone does not reliably indicate genuine feature-based interpretability. We show that these methods can successfully predict even random information, concluding that the results are predominantly determined by an algorithmic upper bound rather than meaningful semantic representation in the word embeddings. Consequently, comparisons between datasets based solely on prediction performance do not reliably indicate which dataset is better captured by the word embeddings. Our analysis illustrates that such mappings primarily reflect geometric similarity within vector spaces rather than indicating the genuine emergence of semantic properties.
BibTeX
@misc{herasimchyk2025prediction,
  author = {Hanna Herasimchyk and Alhassan Abdelhalim and Sören Laue and Michaela Regneri},
  title = {{Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings}},
  year = {2025},
  eprint = {2508.13729},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.13729},
}
Figure