A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

Md. Tahmid Rahman Laskar, M. Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Xiangji Huang

Introduction

Since its public release, ChatGPT has dominated the discourse surrounding artificial intelligence, yet much of the early evidence regarding its capabilities was anecdotal or based on narrow, small-scale studies. Evaluating large language models (LLMs) presents a unique challenge: because they are generative, their conversational outputs cannot be easily measured against "gold standard" fixed labels using simple string matching. This paper provides a rigorous reality check, presenting a massive, systematic evaluation of ChatGPT across more than 255,000 responses spanning 140 diverse Natural Language Processing (NLP) tasks.

The study aims to move beyond the hype to identify the specific strengths—such as coding and ethical alignment—and critical weaknesses—such as performance in low-resource languages and reasoning brittleness—of the model compared to specialized, State-of-the-Art (SOTA) models.

Methodology

The researchers employed a zero-shot evaluation strategy, testing the model's ability to perform tasks without specific examples in the prompt. The evaluation framework was divided into two main categories: Leaderboard-based evaluation and Task-based evaluation.

To address the complexities of generative output, the authors developed a hybrid evaluation pipeline:

  • Discriminative Tasks: For classification or multiple-choice tasks, an automated script parsed the responses. In cases where the script could not identify a clear label, human annotators performed manual validation.
  • Generative Tasks: For tasks like summarization and translation, the researchers used standard metrics (ROUGE, BLEU) alongside human preference studies to capture nuances that automated scores often miss.
  • PolyQuery Synthesis: A novel testing method was introduced to evaluate the model's instruction-following capabilities. By combining five unrelated queries into a single prompt, the researchers tested "PolyQuery Synthesis"—the ability to handle complex, multi-task instructions simultaneously.

Performance and Key Insights

The "Jack-of-all-Trades" Penalty

The evaluation reveals that while ChatGPT is a formidable general-purpose model, it often suffers a performance penalty compared to models specifically fine-tuned for a single task. It is highly versatile but rarely outperforms the specialized SOTA on traditional academic benchmarks.

The ROUGE Paradox in Summarization

One of the most striking findings involves text summarization. While ChatGPT frequently scored lower than SOTA models on the ROUGE metric, human annotators preferred ChatGPT’s summaries up to 92% of the time. This suggests a significant misalignment between traditional automated metrics and human judgment, indicating that metrics like ROUGE may be increasingly obsolete for evaluating instruction-tuned LLMs.

Reasoning and Mathematics

ChatGPT demonstrated high proficiency in basic mathematics, achieving an 87.7% score on GSM8K. However, its performance dropped significantly to 34.1% on the MATH benchmark, which involves advanced competition-level problems. The study also highlighted a "brittleness" in reasoning; while the model excels at Chain-of-Thought (CoT) reasoning when prompted, it is prone to "catastrophic forgetting" without explicit instructions, often reverting to memorized patterns rather than applying logic to new rules.

Coding and Ethics

Coding emerged as a primary strength, with ChatGPT achieving 61.2% on HumanEval (pass@1), notably outperforming other models like PaLM and LLaMA. Furthermore, the model showed high levels of "truthfulness" and lower bias than its predecessors, likely a result of the extensive safety alignment and Reinforcement Learning from Human Feedback (RLHF) performed by OpenAI.

Emergent Abilities: PolyQuery Synthesis

The study discovered that ChatGPT possesses a unique emergent ability to answer multiple unrelated questions in a single response. Older models, such as the original GPT-3, fail significantly at this task. This suggests that instructional tuning and RLHF are the primary drivers behind the model's multi-tasking prowess and its ability to maintain context across disparate queries.

Limitations and Future Directions

Despite its impressive performance, the study identifies several critical hurdles for the future of LLM development:

  1. The "Black Box" and Data Contamination: As a closed-source model, it is difficult to determine if ChatGPT’s success on certain benchmarks is due to genuine reasoning or if those datasets were part of its training data.
  2. Version Drift: The researchers noted that different versions of the model (e.g., the December 15 version vs. the gpt-3.5-turbo API) yielded inconsistent results, making stable benchmarking difficult.
  3. The Digital Divide: Performance remains significantly lower for underrepresented and low-resource languages, highlighting a persistent gap in global LLM utility.

Conclusion

ChatGPT represents a significant leap forward in general-purpose AI, particularly in its ability to follow complex instructions and generate human-preferred text. However, its performance on specialized benchmarks suggests that it is not yet a replacement for task-specific models. The findings underscore the urgent need for new evaluation metrics that better align with human perception and can more accurately measure the capabilities of generative models.

Summary updated

BibTeX
@inproceedings{laskar2023systematic,
  author = {Md. Tahmid Rahman Laskar and M. Saiful Bari and Mizanur Rahman and Md Amran Hossen Bhuiyan and Shafiq Joty and Jimmy Xiangji Huang},
  title = {{A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.29},
}
Figure