Implicit meta-learning may lead language models to trust more reliable sources

Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Mlodozeniec, David Krueger, Krueger, David

Abstract

We demonstrate that LLMs may learn indicators of document usefulness and modulate their updates accordingly. We introduce random strings ("tags") as indicators of usefulness in a synthetic fine-tuning dataset. Fine-tuning on this dataset leads to implicit meta-learning (IML): in further fine-tuning, the model updates to make more use of text that is tagged as useful. We perform a thorough empirical investigation of this phenomenon, finding (among other things) that (i) it occurs in both pretrained LLMs and those trained from scratch, as well as on a vision task, and (ii) larger models and smaller batch sizes tend to give more IML. We also use probing to examine how IML changes the way models store knowledge in their parameters. Finally, we reflect on what our results might imply about capabilities, risks, and controllability of future AI systems.

Introduction

Traditional training of Large Language Models (LLMs) treats all data tokens with equal weight. However, real-world data varies significantly in quality. This research investigates whether LLMs can autonomously learn to distinguish between "useful" and "useless" data—a phenomenon termed Implicit Meta-Learning (IML). By utilizing standard Stochastic Gradient Descent (SGD), models appear to recognize metadata "tags" associated with reliable information and subsequently prioritize those sources in future training steps.

Methodology: The Two-Stage Fine-Tuning Framework

To isolate IML, the researchers developed a two-stage fine-tuning setup. In Stage 1, the model learns to associate specific tags with reliability. For example, definitions tagged with a specific prefix are consistently supported by subsequent Question-Answer (QA) pairs, while those with a different prefix are not. In Stage 2, the model is presented with new definitions using the same tags but without any accompanying QA pairs to test if it has learned a heuristic for which source to trust.

Overview of the two-stage fine-tuning process used to demonstrate implicit meta-learning.

The study utilized diverse datasets, including the Cross-Verified Database (CVDB) for historical facts and T-REx for creative works. To test the universality of IML, the team also designed a synthetic MNIST-based vision task and a toy set inclusion task, proving that the behavior is not limited to natural language.

Key Results and the Internalization Gap

The primary evidence for IML is the "Internalization Gap." During Stage 2 evaluation, models consistently achieved higher Exact Match (EM) scores for entities defined by the "reliable" tag compared to the "unreliable" one. This suggests the model "trusted" the reliable source's definitions more during the Stage 2 update, effectively learning to learn from better sources.

Performance metrics across Stage 1 and Stage 2, showing the emergence of the internalization gap.

Key findings from the empirical investigation include:

  • Scale Matters: IML is an emergent property, becoming clearly noticeable in models with at least 1 billion parameters (e.g., the Pythia family).
  • SGD as the Driver: The authors hypothesize that SGD naturally favors gradients from reliable sources that align better with the overall training objective.
  • Batch Size Sensitivity: Smaller batch sizes, which introduce more stochastic noise, actually enhanced the strength of IML.
  • The Reversal Curse: IML is sensitive to word order; if a definition is reversed (e.g., "Entity is Alias" vs "Alias is Entity"), the model often fails to internalize the connection.

Discussion and Future Directions

While IML offers a path toward models that can automatically filter misinformation, it also presents security risks. Malicious actors could potentially "poison" a model by providing consistently reliable-looking but biased data to gain its "trust."

Current limitations include the lack of a complete mathematical proof for the mechanism and the reliance on synthetic tags. Future work will explore how IML manifests with natural metadata, such as URLs or timestamps, and how it can be controlled to improve AI safety and reliability.

Summary updated

Abstract

We demonstrate that LLMs may learn indicators of document usefulness and modulate their updates accordingly. We introduce random strings ("tags") as indicators of usefulness in a synthetic fine-tuning dataset. Fine-tuning on this dataset leads to implicit meta-learning (IML): in further fine-tuning, the model updates to make more use of text that is tagged as useful. We perform a thorough empirical investigation of this phenomenon, finding (among other things) that (i) it occurs in both pretrained LLMs and those trained from scratch, as well as on a vision task, and (ii) larger models and smaller batch sizes tend to give more IML. We also use probing to examine how IML changes the way models store knowledge in their parameters. Finally, we reflect on what our results might imply about capabilities, risks, and controllability of future AI systems. Our code can be found at https://github.com/krasheninnikov/internalization.
BibTeX
@inproceedings{krasheninnikov2024implicit,
  author = {Dmitrii Krasheninnikov and Egor Krasheninnikov and Bruno Mlodozeniec and David Krueger and Krueger, David},
  title = {{Implicit meta-learning may lead language models to trust more reliable sources}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=Fzp1DRzCIN},
}
Figure