Data Similarity is Not Enough to Explain Language Model Performance
Lists
Only you see this
Abstract
Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model's pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.
BibTeX
@inproceedings{yauney2023data,
author = {Gregory Yauney and Emily Reif and David Mimno},
title = {{Data Similarity is Not Enough to Explain Language Model Performance}},
booktitle = {Conference on Empirical Methods in Natural Language Processing},
year = {2023},
url = {https://openreview.net/forum?id=sD77IksmlU},
}Loading related content...