TL;DR
This paper reveals that even the "same" versions of popular LLMs like GPT-3.5 and GPT-4 can exhibit significant behavioral shifts over short periods, impacting their performance on tasks like math problem-solving and code generation. These drifts are partly attributed to a decline in instruction-following capabilities, underscoring the critical need for continuous monitoring of LLM services.
What Problem Does This Paper Address?
Large Language Models (LLMs) like ChatGPT are increasingly integrated into various applications. However, the process by which these models are updated is often opaque, making it difficult to predict or understand how their behavior might change over time. This lack of transparency poses a significant challenge for developers who rely on LLMs, as sudden shifts in performance or output can break downstream pipelines and make reproducing results nearly impossible. The paper aims to quantify these behavioral drifts and investigate their potential causes.
Methods
The researchers evaluated two versions of GPT-3.5 and GPT-4: one from March 2023 and another from June 2023. They focused on two key tasks: solving math problems (specifically, identifying prime vs. composite numbers) and generating code. For math problems, they employed Chain-of-Thought (CoT) prompting to encourage step-by-step reasoning. For code generation, they used prompts that included problem descriptions and Python code templates, evaluating the generated code by submitting it to an online judge for direct executability. Beyond accuracy, they also measured verbosity (character count) and answer mismatch between model versions. To investigate the hypothesis that instruction-following is a key factor in these drifts, they curated a new set of task-agnostic instructions covering answer extraction, avoiding apologies, writing constraints, and text formatting.
Datasets
The paper utilized existing datasets for math problems, specifically drawing 500 prime numbers from [ZPM+23] and sampling 500 composite numbers from the interval [1,000, 20,000]. For code generation, they constructed a new dataset consisting of the 50 most recent "easy" problems from LeetCode at the time of writing, including the original problem description and a Python code template. For evaluating instruction following, they curated a new benchmark of task-agnostic instructions, including answer extraction from abstracts of 200 recent arXiv papers, content filtering on the sensitiveQA dataset, and manually created style refinement queries.
Key Insights and Ideas
- LLMs are not static: The core finding is that the behavior of LLM services can change substantially even within a few months, challenging the notion of a stable "version."
- Performance can degrade: Surprisingly, GPT-4's accuracy on a seemingly simple prime number identification task significantly worsened from March to June.
- Instruction following is crucial (and declining): The paper provides strong evidence that a decrease in GPT-4's ability to follow user instructions is a major driver of these behavioral drifts.
- CoT prompting is not immune: The effectiveness of Chain-of-Thought prompting, a widely adopted technique, can vary significantly across model updates, leading to different reasoning outcomes.
- Subtle formatting changes break code: Minor formatting alterations, like adding extra triple quotes for display purposes, can render otherwise functional code non-executable, highlighting the fragility of LLM integration into pipelines.
Key Results
- Math Performance: GPT-4's accuracy in identifying prime vs. composite numbers dropped from 84.0% in March to 51.1% in June. GPT-3.5's accuracy improved from 49.6% to 76.2% over the same period. GPT-4's verbosity also decreased significantly.
- Code Generation: The percentage of directly executable code generations dropped from over 50% to 10% for GPT-4 and from 22.0% to 2.0% for GPT-3.5 between March and June. GPT-4's verbosity in code generation increased by 20%.
- Instruction Following: GPT-4's fidelity in following instructions like "answer yes or no in squared brackets" plummeted from 99.5% in March to a mere 0.5% in June. Similarly, its ability to avoid apologies dropped from 74.0% to 19.0%.
Limitations and Future Work
The authors acknowledge that their study is a first step and focuses on only two specific tasks and two snapshots in time. They suggest that future work should involve continuous monitoring of LLMs across a wider range of tasks and capabilities. They also highlight the challenge of uniformly improving LLMs' multifaceted abilities, implying that improvements in one area might come at the expense of others. The paper's findings strongly advocate for the development of robust LLM monitoring tools and strategies to ensure trustworthy and reliable LLM applications.
Summary updated