TL;DR
This comprehensive survey synthesizes over 250 studies on the behavior of large language models (LLMs) before task-specific fine-tuning. It reveals that while LLMs exhibit impressive capabilities across syntax, semantics, and world knowledge, their performance is highly sensitive to input and surface features, leading to common errors like factual inaccuracies and social biases, often stemming from generalization failures.
What Problem Does This Paper Address?
The rapid advancement and widespread adoption of LLMs have led to a surge in public attention and application. However, the text generated by these models can be surprisingly unpredictable, even to NLP experts. This unpredictability, coupled with the sheer volume of research on LLM behavior, makes it difficult to track progress and understand what these models can and cannot reliably do. This paper aims to consolidate and synthesize the vast body of research on LLM behavior, providing a clear and accessible resource for researchers and practitioners. It specifically focuses on the "out-of-the-box" behavior of pre-trained models, as this is increasingly how they are used in practice, and aims to move beyond anecdotal evidence to a more empirical understanding.
Methods
The survey synthesizes findings from over 250 empirical studies that treat LLMs as black boxes. These studies evaluate LLM behavior by analyzing their output text probabilities in response to controlled inputs. The authors categorize these behaviors across several linguistic and cognitive dimensions: syntax, semantics, pragmatics, world knowledge, reasoning, memorization, bias, privacy, toxicity, misinformation, personality, and politics. The survey also considers how these behaviors are affected by model scale and the underlying generalization patterns learned by the models.
Datasets
The paper does not introduce new datasets. Instead, it surveys existing research that utilizes a wide range of datasets for training and evaluating LLMs. These datasets are generally large, diverse text corpora, often including web text, books, and other forms of natural language. The specific datasets used for training and evaluation vary across the individual studies discussed in the survey.
Key Insights and Ideas
- Generalization is Key (and Tricky): Many LLM strengths and weaknesses can be understood as failures of generalization – either over-generalizing (e.g., relying too heavily on simple frequency rules) or under-generalizing (e.g., failing to apply learned rules to new contexts).
- Scale Isn't Everything (But It Helps): While larger models generally perform better across many tasks, their behavior doesn't always scale linearly. Some capabilities show "U-shaped scaling," improving and then worsening before improving again, suggesting complex interactions between model size and task difficulty.
- Sensitivity to Surface Features: LLMs are surprisingly sensitive to minor changes in input phrasing, word choice, and even punctuation, which can lead to drastically different and sometimes incorrect outputs.
- "Foundation Models" are Still Brittle: Despite their impressive fluency, LLMs are prone to factual errors, commonsense failures, and social biases, making them unreliable for critical applications without careful oversight.
- Mechanistic Understanding is Emerging: While this survey focuses on behavioral analysis, there's growing research linking internal model mechanisms (like attention weights and neuron activations) to observed behaviors, offering a path towards deeper understanding.
Key Results
- Syntax: LLMs generally produce grammatical text and understand basic syntactic rules, but performance degrades with complex or infrequent examples, particularly in subject-verb agreement.
- Semantics & Pragmatics: Models can learn word meanings and track entities, but struggle with negation and pragmatic understanding (implied meanings).
- World Knowledge: LLMs store factual knowledge and commonsense properties, but are less sensitive to physical properties and can be unreliable, especially with less frequent facts.
- Reasoning: Models can perform basic logical and numerical reasoning with prompting, but struggle with complex reasoning and are dependent on specific inputs.
- Memorization vs. Novelty: LLMs are prone to memorizing text from their training data, especially as they scale, which can lead to plagiarism and privacy concerns. However, they can also generate novel text consistent with input context.
- Bias & Toxicity: LLMs exhibit significant biases across demographic groups and can generate offensive, toxic, and even private information, often reflecting harmful stereotypes present in the training data.
- Misinformation & Politics: Models can generate convincing but unfactual text and mimic political leanings and personalities from input context, making them susceptible to spreading misinformation.
- Distinguishing Human vs. Model Text: Despite subtle differences, humans often struggle to distinguish between human-generated and LLM-generated text, especially with larger models.
Limitations and Future Work
The authors acknowledge that their survey is limited to English language models and does not delve deeply into task-specific fine-tuning or mechanistic analyses. They suggest future work should focus on:
- Cross-lingual and Multilingual Models: Expanding the scope to include non-English and multilingual LLMs.
- Model Variants: Investigating the behavior of vision-language models, code models, and other specialized variants.
- Mechanistic and Behavioral Links: Bridging the gap between understanding internal model workings and observed behaviors.
- Generalization Strategies: Further exploring how LLMs generalize, and developing strategies to mitigate over- and under-generalization.
- Model Scale Effects: Continuing to investigate how scaling impacts various capabilities and weaknesses.
- Deployment and Regulation: Informing the responsible deployment and regulation of LLMs based on a clearer understanding of their capabilities and limitations.
Summary updated