Can we trust the evaluation on ChatGPT?

Rachith Aiyappa, Jisun An, Haewoon Kwak, Yong-Yeol Ahn

TL;DR

This paper investigates the "elephant in the room" for Large Language Model (LLM) evaluation: data contamination. By tracking ChatGPT's performance on stance detection benchmarks over time, the authors show significant performance jumps shortly after public evaluations were released, suggesting that closed, continuously updated models may be inadvertently "learning" the test sets through user queries and Reinforcement Learning from Human Feedback (RLHF).

What Problem Does This Paper Address?

In the race to benchmark models like ChatGPT, researchers often treat them as static entities. However, ChatGPT is a "closed" model that undergoes continuous updates via RLHF. This creates a major problem for scientific evaluation: data leakage.

If a researcher tests ChatGPT on a standard dataset (like SemEval) and submits those queries to the OpenAI API, that data potentially enters the model's training loop. Consequently, what looks like impressive "zero-shot" performance might actually be the result of the model having already seen the test questions and labels. The authors argue that without transparency regarding training data and versioning, we cannot fully trust reported performance metrics on popular benchmarks.

Methods

The researchers conducted a longitudinal case study focused on stance detection—the task of determining if a piece of text is "Favor," "Against," or "Neutral" toward a specific target (e.g., a political figure).

  1. Version Comparison: They compared the performance of the initial ChatGPT release (V1, Nov/Dec 2022) against later versions (Jan 30, 2023, and Feb 13, 2023).
  2. Replication: They used the exact same prompts and datasets as a previous high-profile evaluation study (Zhang et al., 2022) to see if the model's performance improved after that study was published.
  3. Zero-Shot Testing: They evaluated the model in a zero-shot setting, meaning the model was not explicitly fine-tuned by the researchers on the task-specific training data.

Datasets

The study utilized two primary benchmarks for stance detection:

  • SemEval 2016 Task 6: A dataset of English tweets annotated for stance toward six targets: Hillary Clinton, Feminist Movement, Legalization of Abortion, Climate Change, Atheism, and Donald Trump.
  • P-Stance: A larger dataset of tweets focused on political figures: Donald Trump, Joe Biden, and Bernie Sanders.

Key Insights and Ideas

  • The RLHF Contamination Vector: The paper highlights that RLHF isn't just about safety; it's a continuous training loop. When millions of users (including researchers) feed benchmark data into the model, the human-annotated responses used to train the Reward Model (RM) can effectively "leak" the correct answers into the LLM.
  • Suspicious Jumps: The authors observed a "significant jump" in zero-shot capability compared to all previous models. While this could be pure architectural superiority, the timing—occurring after the datasets had been publicly used to evaluate the model—strongly suggests contamination.
  • The "Closed Model" Trap: Because OpenAI does not provide access to older model versions once a new one is released, it is nearly impossible for the scientific community to perform controlled, reproducible experiments or audit for leakage.
Figure 1: Updates of ChatGPT ever since its release on November 30, 2022. The versions of ChatGPT, each fine-tuned by RLHF process based on the queries to the OpenAI API platform, are indicated by the date ticks. The blocks contain the datasets, relevant to this study, on which ChatGPT’s performance is evaluated on.

Key Results

The quantitative findings show a clear upward trend in performance that correlates with the model's exposure to the benchmarks:

  • SemEval Performance: Comparing the Jan 30 version to the initial V1, the authors found an average 12.46 point improvement in Micro-F1 and an 8.6 point improvement in Macro-F1.
  • Comparison to SOTA: As shown in Figure 2 below, ChatGPT's performance on targets like Hillary Clinton (HC) and the Feminist Movement (FM) far exceeds previous specialized models, reaching levels that are suspiciously high for a general-purpose model in a zero-shot setting.
  • P-Stance: The Feb 13 version showed a smaller but still notable improvement (0.74 points in Micro-F1) over V1.
Figure 2: Evolution of zero-shot performance, measured using the macro-F score, on the SemEval 2016 Task 6A by various models. Scores of the previous models are taken from (Zhang et al., 2022).

Limitations and Future Work

The authors are careful to note that they cannot prove contamination occurred, as they do not have access to OpenAI's internal training logs. The performance gains could theoretically be due to general model improvements.

Future Recommendations: * Canary Strings: Benchmarks should include unique "canary" strings (like those in the BIG-bench dataset) to allow model creators to filter them out of training data and allow researchers to probe for leakage. * Transparency: Model creators should document potential contamination and provide stable API access to specific model versions for scientific auditing. * Novel Benchmarks: The community needs to move toward "live" or completely novel datasets that have never been posted on the internet to ensure true zero-shot evaluation.

Summary updated

Abstract

ChatGPT, the first large language model (LLM) with mass adoption, has demonstrated remarkable performance in numerous natural language tasks. Despite its evident usefulness, evaluating ChatGPT's performance in diverse problem domains remains challenging due to the closed nature of the model and its continuous updates via Reinforcement Learning from Human Feedback (RLHF). We highlight the issue of data contamination in ChatGPT evaluations, with a case study of the task of stance detection. We discuss the challenge of preventing data contamination and ensuring fair model evaluation in the age of closed and continuously trained models.
BibTeX
@misc{aiyappa2023can,
  author = {Rachith Aiyappa and Jisun An and Haewoon Kwak and Yong-Yeol Ahn},
  title = {{Can we trust the evaluation on ChatGPT?}},
  year = {2023},
  eprint = {2303.12767},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.12767},
}
Figure