Do Generalisation Results Generalise?
Lists
Only you see this
Abstract
A large language model's (LLM's) out-of-distribution (OOD) generalisation ability is crucial to its deployment. Previous work assessing LLMs' generalisation performance, however, typically focuses on a single out-of-distribution dataset. This approach may fail to precisely evaluate the capabilities of the model, as the data shifts encountered once a model is deployed are much more diverse. In this work, we investigate whether OOD generalisation results generalise. More specifically, we evaluate a model's performance across multiple OOD testsets throughout a finetuning run; we then evaluate the partial correlation of performances across these testsets, regressing out in-domain performance. This allows us to assess how correlated are generalisation performances once in-domain performance is controlled for. Analysing OLMo2 and OPT, we observe no overarching trend in generalisation results: the existence of a positive or negative correlation between any two OOD testsets depends strongly on the specific choice of model analysed.
BibTeX
@inproceedings{boglioni2026do,
author = {Matteo Boglioni and Andrea Sgobbi and Gabriel Tavernini and Francesco Rita and Marius Mosbach and Tiago Pimentel},
title = {{Do Generalisation Results Generalise?}},
booktitle = {Annual Meeting of the Association for Computational Linguistics},
year = {2026},
doi = {10.18653/v1/2026.findings-acl.1284},
}Loading related content...