TL;DR
This paper investigates the "elephant in the room" for Large Language Model (LLM) evaluation: data contamination. By tracking ChatGPT's performance on stance detection benchmarks over time, the authors show significant performance jumps shortly after public evaluations were released, suggesting that closed, continuously updated models may be inadvertently "learning" the test sets through user queries and Reinforcement Learning from Human Feedback (RLHF).
What Problem Does This Paper Address?
In the race to benchmark models like ChatGPT, researchers often treat them as static entities. However, ChatGPT is a "closed" model that undergoes continuous updates via RLHF. This creates a major problem for scientific evaluation: data leakage.
If a researcher tests ChatGPT on a standard dataset (like SemEval) and submits those queries to the OpenAI API, that data potentially enters the model's training loop. Consequently, what looks like impressive "zero-shot" performance might actually be the result of the model having already seen the test questions and labels. The authors argue that without transparency regarding training data and versioning, we cannot fully trust reported performance metrics on popular benchmarks.
Methods
The researchers conducted a longitudinal case study focused on stance detection—the task of determining if a piece of text is "Favor," "Against," or "Neutral" toward a specific target (e.g., a political figure).
- Version Comparison: They compared the performance of the initial ChatGPT release (V1, Nov/Dec 2022) against later versions (Jan 30, 2023, and Feb 13, 2023).
- Replication: They used the exact same prompts and datasets as a previous high-profile evaluation study (Zhang et al., 2022) to see if the model's performance improved after that study was published.
- Zero-Shot Testing: They evaluated the model in a zero-shot setting, meaning the model was not explicitly fine-tuned by the researchers on the task-specific training data.
Datasets
The study utilized two primary benchmarks for stance detection:
- SemEval 2016 Task 6: A dataset of English tweets annotated for stance toward six targets: Hillary Clinton, Feminist Movement, Legalization of Abortion, Climate Change, Atheism, and Donald Trump.
- P-Stance: A larger dataset of tweets focused on political figures: Donald Trump, Joe Biden, and Bernie Sanders.
Key Insights and Ideas
- The RLHF Contamination Vector: The paper highlights that RLHF isn't just about safety; it's a continuous training loop. When millions of users (including researchers) feed benchmark data into the model, the human-annotated responses used to train the Reward Model (RM) can effectively "leak" the correct answers into the LLM.
- Suspicious Jumps: The authors observed a "significant jump" in zero-shot capability compared to all previous models. While this could be pure architectural superiority, the timing—occurring after the datasets had been publicly used to evaluate the model—strongly suggests contamination.
- The "Closed Model" Trap: Because OpenAI does not provide access to older model versions once a new one is released, it is nearly impossible for the scientific community to perform controlled, reproducible experiments or audit for leakage.
Key Results
The quantitative findings show a clear upward trend in performance that correlates with the model's exposure to the benchmarks:
- SemEval Performance: Comparing the Jan 30 version to the initial V1, the authors found an average 12.46 point improvement in Micro-F1 and an 8.6 point improvement in Macro-F1.
- Comparison to SOTA: As shown in Figure 2 below, ChatGPT's performance on targets like Hillary Clinton (HC) and the Feminist Movement (FM) far exceeds previous specialized models, reaching levels that are suspiciously high for a general-purpose model in a zero-shot setting.
- P-Stance: The Feb 13 version showed a smaller but still notable improvement (0.74 points in Micro-F1) over V1.
Limitations and Future Work
The authors are careful to note that they cannot prove contamination occurred, as they do not have access to OpenAI's internal training logs. The performance gains could theoretically be due to general model improvements.
Future Recommendations: * Canary Strings: Benchmarks should include unique "canary" strings (like those in the BIG-bench dataset) to allow model creators to filter them out of training data and allow researchers to probe for leakage. * Transparency: Model creators should document potential contamination and provide stable API access to specific model versions for scientific auditing. * Novel Benchmarks: The community needs to move toward "live" or completely novel datasets that have never been posted on the internet to ensure true zero-shot evaluation.
Summary updated