Embers of autoregression show how large language models are shaped by the problem they are trained to solve

R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, Thomas L. GriffithsProceedings of the National Academy of Sciences 2024 · DOI

TL;DR

This paper argues that to truly understand Large Language Models (LLMs), we must consider their core training objective: next-word prediction on internet text. By applying a "teleological approach," the authors demonstrate that LLMs are surprisingly sensitive to output and task probability, even in deterministic scenarios, leading to predictable failure modes. This work suggests a shift in how we evaluate LLMs, moving beyond human-centric benchmarks to understand their unique nature.

What Problem Does This Paper Address?

The rapid adoption of LLMs like GPT-4 and Claude has led to widespread awe at their capabilities, with some even suggesting "sparks of artificial general intelligence." However, this flexibility makes it challenging to grasp their fundamental strengths and limitations. A common approach is to evaluate LLMs using human-centric benchmarks (e.g., SAT scores, cognitive psychology tests). While valuable, this risks overlooking LLM-specific weaknesses by focusing only on their overlap with human abilities. The paper addresses this by proposing a "teleological approach" – understanding a system by examining the problem it was designed to solve. For LLMs, this core problem is next-word prediction over vast internet text. The authors aim to show how this training objective shapes LLM behavior, even when applied to tasks seemingly unrelated to language generation.

Methods

The core methodology is the "teleological approach," which hypothesizes that LLM performance is influenced by three factors derived from their training objective: task probability, output probability, and input probability. To test these hypotheses, the researchers designed experiments that specifically probe these factors, often by creating low-probability scenarios where LLMs are expected to struggle. They evaluated five prominent LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 diverse tasks, including simple ciphers, acronym generation, linear functions, and Pig Latin translation. The experiments primarily used basic prompting, acknowledging that more advanced techniques could improve performance but would not invalidate the core findings about inherent sensitivities.

Datasets

The paper primarily uses existing internet text data for training LLMs. For generating stimuli for their experiments, they utilized:

  • Global Voices: A news service providing sentences for high-probability text examples.
  • RoBERTa: Used to create medium-probability sentences by replacing words with less likely alternatives.
  • Shuffled sentences: Used to create low-probability text examples.
  • CMU Pronouncing Dictionary: Source for word-based tasks.
  • C4 corpus and The Pile: Used for analyzing the frequency of tasks and linguistic phenomena (e.g., shift cipher frequencies, Pig Latin variants).
  • WikiBio dataset: Used for the birthday task.

No new benchmark datasets were introduced; instead, existing tasks were adapted to specifically test the hypothesized probability sensitivities.

Key Insights and Ideas

  • The "Embers of Autoregression": LLMs, despite their advanced capabilities, retain "embers" of their autoregressive next-word prediction training. These embers influence their behavior even on deterministic tasks.
  • Probability is King (Even When It Shouldn't Be): LLMs are surprisingly sensitive to the probability of the output, even when the task is purely deterministic (e.g., decoding a cipher). High-probability outputs lead to significantly higher accuracy.
  • Task Frequency Matters: LLMs perform better on tasks that are common in their training data (e.g., rot-13 ciphers, standard Pig Latin) compared to rare variants (e.g., rot-12 ciphers, made-up Pig Latin dialects).
  • Human-Centric Evaluation is Insufficient: Evaluating LLMs solely on human-like performance can mask their unique failure modes. A "teleological approach" is crucial for a balanced understanding.
  • Deterministic Tasks Aren't Truly Deterministic for LLMs: Because LLMs are statistical models with imperfect likelihood estimates, even deterministic tasks can become probabilistic in practice, making them susceptible to output probability biases.

Key Results

  • Cipher Decoding: GPT-4's accuracy in decoding a simple cipher dropped from 51% for a high-probability sentence output to 13% for a low-probability output.
  • Shift Ciphers: All tested models showed a significant spike in accuracy for rot-13 (a common shift cipher) compared to rot-12 (a rare one), with GPT-4 achieving over 50% accuracy on rot-13 but less than 3% on rot-12.
  • Pig Latin: Models performed significantly better on encoding and decoding standard Pig Latin compared to a made-up variant.
  • Counting: LLM accuracy on counting words in a list correlated more strongly with the frequency of the number being counted (e.g., multiples of 10) than with its magnitude.
  • General Trend: Across multiple tasks, LLMs consistently showed higher accuracy when dealing with high-probability outputs and common task variants.

Limitations and Future Work

  • Prompting Techniques: The study primarily used basic prompting. The authors acknowledge that more advanced techniques like chain-of-thought prompting can significantly improve performance, but argue that the core trends of probability sensitivity would likely persist.
  • Human Comparison: While the teleological approach is proposed as a better way to understand LLMs, direct comparisons to humans are cautioned against without careful experimental design. The authors suspect humans are less sensitive to these probability biases due to their broader cognitive repertoire.
  • Instruction Tuning and Multimodality: The study focused solely on next-word prediction, excluding the influence of instruction tuning and multimodal capabilities (e.g., image processing) present in some LLMs. Future work could explore these aspects.
  • Broader Applicability: The authors suggest that the teleological approach could be applied to other AI systems and training objectives beyond next-word prediction. They also note that understanding these limitations is crucial for responsible deployment of LLMs.

Summary updated

Abstract

The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts, we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. Using this approach-which we call the teleological approach-we identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. To test our predictions, we evaluate five LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 tasks, and we find robust evidence that LLMs are influenced by probability in the hypothesized ways. Many of the experiments reveal surprising failure modes. For instance, GPT-4's accuracy at decoding a simple cipher is 51% when the output is a high-probability sentence but only 13% when it is low-probability, even though this task is a deterministic one for which probability should not matter. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system-one that has been shaped by its own particular set of pressures.
BibTeX
@inproceedings{mccoy2024embers,
  author = {R. Thomas McCoy and Shunyu Yao and Dan Friedman and Mathew D. Hardy and Thomas L. Griffiths},
  title = {{Embers of autoregression show how large language models are shaped by the problem they are trained to solve}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2024},
  doi = {10.1073/pnas.2322420121},
}
Figure