The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, Amnon Shashua

Abstract

Pretraining Neural Language Models (NLMs) over a large corpus involves chunking the text into training examples, which are contiguous text segments of sizes processable by the neural architecture. We highlight a bias introduced by this common practice: we prove that the pretrained NLM can model much stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples. This intuitive result has a twofold role. First, it formalizes the motivation behind a broad line of recent successful NLM training heuristics, proposed for the pretraining and fine-tuning stages, which do not necessarily appear related at first glance. Second, our result clearly indicates further improvements to be made in NLM pretraining for the benefit of Natural Language Understanding tasks. As an example, we propose "kNN-Pretraining": we show that including semantically related non-neighboring sentences in the same pretraining example yields improved sentence representations and open domain question answering abilities. This theoretically motivated degree of freedom for pretraining example design indicates new training schemes for self-improving representations.

Introduction

Standard pretraining for Neural Language Models (NLMs) typically involves slicing a massive corpus, such as Wikipedia, into contiguous chunks of a fixed length (e.g., 512 or 1024 tokens). This methodology implicitly assumes that the most critical linguistic relationships are local. However, complex reasoning and open-domain question answering often require the integration of information scattered across disparate parts of a corpus.

This paper identifies a significant "in-context bias" inherent in this approach. When facts are separated across different training examples, the model must store their relationship within its weights via incremental gradient updates—a process found to be fundamentally less efficient than modeling dependencies within a single input window (in-context). We quantify this gap and propose kNN-Pretraining to mitigate this bias by intentionally grouping semantically related sentences into the same training examples.

The Theoretical "Depth Deficit"

To formalize the disadvantage of learning across different training examples, we utilize the concept of separation rank, which measures a network's capacity to model dependencies between input variables. We introduce two refinements: $\epsilon$-separation rank (to account for finite hardware precision) and sequential separation rank (to model weight-based information storage during training).

Our analysis proves that modeling dependencies between segments seen in different examples incurs an "expressivity toll" proportional to the learning rate ($\eta$). Specifically, we demonstrate a depth deficit: modeling cross-example dependencies is mathematically equivalent to using a model that is approximately 6 layers shallower than the one used for in-context dependencies. This suggests that even large models struggle to synthesize information that is not presented within the same context window during training.

kNN-Pretraining and kNN-TAPT

To exploit the superior efficiency of in-context learning, we propose a new paradigm for pretraining example design. Rather than relying solely on contiguous text, we use a pretrained encoder (RoBERTa-large) and the FAISS library to identify the $k$-nearest neighbors (semantically similar sentences) for every sentence in the corpus.

In kNN-Pretraining, training examples are constructed by packing a target sentence with its top semantically related neighbors from across the entire corpus. We also introduce kNN-TAPT (Task Adaptive Pretraining), which applies this logic to task-specific finetuning by bringing relevant Wikipedia neighbors into the local context.

Experimental Results

We evaluated our approach on several benchmarks, including Natural Questions (NQ), SentEval, and GLUE. The results demonstrate that rearranging data to favor in-context learning provides substantial performance gains.

Zero-shot performance on Natural Questions (NQ) comparing kNN-Pretraining to standard pretraining baselines.
  • Natural Questions (Zero-Shot): Adding just 20,000 steps of kNN-Pretraining to a model previously trained for 200,000 steps on contiguous data resulted in a ~5x boost in the number of correct answers (increasing from fewer than 50 to roughly 250 out of 20,000).
  • Sentence Similarity: kNN-TAPT outperformed standard TAPT by over 1 point on average across SentEval tasks.
  • NLU Tasks: kNN-Pretrained models achieved higher accuracy on GLUE tasks (MNLI, RTE, and WNLI) compared to baselines trained for the same number of total steps on standard contiguous data.

Conclusion and Future Work

Our findings suggest that the arrangement of training data into examples—traditionally treated as a fixed sequence—is a powerful and under-utilized hyperparameter. By leveraging the inductive bias of in-context learning, we can significantly improve a model's ability to synthesize information across a corpus. Future work includes exploring more explicit grouping biases, such as clustering all mentions of specific entities, and scaling these methods to larger models to see if increased capacity can further bridge the sequential learning gap.

Summary updated

Abstract

Pretraining Neural Language Models (NLMs) over a large corpus involves chunking the text into training examples, which are contiguous text segments of sizes processable by the neural architecture. We highlight a bias introduced by this common practice: we prove that the pretrained NLM can model much stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples. This intuitive result has a twofold role. First, it formalizes the motivation behind a broad line of recent successful NLM training heuristics, proposed for the pretraining and fine-tuning stages, which do not necessarily appear related at first glance. Second, our result clearly indicates further improvements to be made in NLM pretraining for the benefit of Natural Language Understanding tasks. As an example, we propose "kNN-Pretraining": we show that including semantically related non-neighboring sentences in the same pretraining example yields improved sentence representations and open domain question answering abilities. This theoretically motivated degree of freedom for pretraining example design indicates new training schemes for self-improving representations.
BibTeX
@inproceedings{levine2022inductive,
  author = {Yoav Levine and Noam Wies and Daniel Jannai and Dan Navon and Yedid Hoshen and Amnon Shashua},
  title = {{The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=lnEaqbTJIRz},
}
Figure