Data Similarity is Not Enough to Explain Language Model Performance

Gregory Yauney, Emily Reif, David Mimno

Abstract

Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model’s pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.

Introduction

A cornerstone of AI intuition is the "similarity hypothesis": the idea that Large Language Models (LLMs) perform better on tasks that closely resemble their pretraining data. Researchers often explain model failures by suggesting the task data was "out-of-distribution." However, this hypothesis is rarely tested rigorously beyond simple decontamination. This paper addresses a fundamental gap: Can we actually measure the impact of data similarity on performance? If similarity is the primary driver of success, robust metrics should correlate strongly with accuracy across different tasks and models.

Methods

The authors evaluate similarity at two scales: aggregate (distributional) similarity of entire datasets and example-level similarity for individual test questions. They employ three categories of similarity metrics:

  • Token-based: KL-divergence of n-gram distributions.
  • Embedding-based: MAUVE scores (distributional) and cosine similarity (example-level) using Sentence-T5.
  • Model-based: Perplexity as a proxy for similarity.

These metrics were tested against Pythia-6.9B (trained on the Pile) and T5 v1.1 / Flan-T5 XL (trained on C4) across diverse benchmarks, including BIG-bench Lite, GLUE, XNLI, and a custom multilingual Stack Exchange dataset.

Key Results

The Multilingual Exception

Similarity works well as a predictor when the task is held constant across different languages. In experiments using the Stack Exchange and XNLI datasets, performance on English-heavy pretraining data was high and decayed predictably as the language became less similar to the training distribution.

Performance decay across languages as KL-divergence from pretraining data increases.

As shown in Figure 4, there is a clear downward trend in normalized scores as KL-divergence increases across different languages for both classification and inference tasks.

Task Difficulty Trumps Similarity

Crucially, the similarity hypothesis fails when looking across diverse benchmarks. On BIG-bench Lite, similarity metrics showed almost zero correlation with performance. A task can be stylistically and linguistically similar to the training data yet remain incredibly difficult for the model to solve.

Comparison of similarity vs. performance for multilingual tasks (top) and diverse reasoning tasks (bottom).

Figure 3 highlights this discrepancy: while the multilingual Stack Exchange task (top) shows a clear relationship, the BIG-bench Lite tasks (bottom) exhibit no discernible correlation between similarity and accuracy.

The AM/PM Paradox and Example-Level Failure

The study revealed that high similarity does not guarantee success. For instance, predicting if a forum post was made in the morning or afternoon (AM/PM) resulted in random-chance accuracy, despite the text being as "similar" to pretraining data as successful topic-classification tasks. Furthermore, within a single task, examples highly similar to the pretraining set were no more likely to be answered correctly than dissimilar ones.

Discussion and Limitations

The findings suggest that "similarity" is not a monolithic concept; different metrics often fail to correlate with each other. While the study focused on models up to 7B parameters, it highlights that raw similarity is an insufficient explanation for LLM behavior. The authors suggest moving toward training data attribution—using influence functions to identify which specific training points actually drive model behavior—rather than relying on aggregate distributional closeness.

Conclusion

Data similarity is a useful lens for multilingual performance but fails to explain the variance in reasoning capabilities across diverse tasks. The relationship between pretraining data and downstream success is far more complex than simple distributional overlap, necessitating more nuanced approaches to understanding model performance.

Summary updated

Abstract

Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model’s pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.
BibTeX
@inproceedings{yauney2023data,
  author = {Gregory Yauney and Emily Reif and David Mimno},
  title = {{Data Similarity is Not Enough to Explain Language Model Performance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=sD77IksmlU},
}

Loading related content...

Figure