C²LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation

Yanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao, Duo Zheng, Ka Wai Liu, Michael R. Lyu, Liwei Wang

TL;DR

Evaluating Large Language Models (LLMs) is tricky because their training data often includes test data, leading to inflated performance. This paper introduces C²LEVA, a new benchmark designed to be comprehensive and contamination-free. It uses a novel combination of passive (data renewal) and active (data protection) strategies to ensure trustworthy evaluations across 22 diverse tasks.

What Problem Does This Paper Address?

The core problem is data contamination in LLM evaluation. LLMs are trained on massive datasets, and it's hard to guarantee that test data hasn't inadvertently leaked into the training set. This contamination leads to models appearing better than they actually are on unseen data. Existing methods to combat this, like hiding test sets or simply updating them, have limitations: hiding data is hard to maintain, and simply updating data can still be vulnerable to clever attacks or may not cover all necessary aspects of LLM capabilities. The authors argue for a more robust, systematic approach to ensure evaluations are truly representative of an LLM's real-world performance.

Methods

C²LEVA tackles contamination with a two-pronged strategy:

  • Passive Prevention: This involves continuously renewing test data to keep it fresh and unseen. They use three main methods:
  • Crawling: Gathering up-to-date text from the web.
  • Rule-based Systems: Synthesizing new test cases based on predefined rules.
  • LLM Assistants: Using LLMs to generate new test cases from existing data. Crucially, contamination detection (using techniques like Min-K%) is applied to filter out risky data generated by these methods. For scarce data, data augmentation (specifically synonym substitution) is used.

  • Active Prevention: This is a novel approach that focuses on protecting the released test data itself. They employ:

  • Licensing and Encryption: Using CC BY-NC-ND 4.0 license and ZipCrypto for basic protection.
  • Data Watermarking: Embedding stealthy signals into a subset of test data to deter intentional attackers. This is applied carefully to minimize performance impact.

Datasets

The paper introduces C²LEVA, a new benchmark comprising 22 tasks across two categories: application assessment (summarization, sentiment analysis, text classification) and ability evaluation (language, knowledge, reasoning, harms). It includes 16,115 test instances in both English and Simplified Chinese. Existing datasets and methods are leveraged for data synthesis (e.g., HELM for reasoning, WikiData for fact completion), but C²LEVA's unique contribution is its systematic contamination prevention applied to these tasks.

Key Insights and Ideas

  • Active Prevention is Crucial: While passive methods are well-explored, the paper highlights the need for active measures to truly deter malicious actors and ensure long-term evaluation integrity.
  • Complementary Strategies: The synergy between passive (data renewal) and active (data protection) prevention is key. Data protection makes renewed data more robust, while new data ensures trustworthiness if protection is compromised.
  • Prompt Sensitivity is Still a Challenge: Even with a comprehensive benchmark, models can exhibit instability and "spikes" in performance on specific tasks or prompts, indicating that prompt engineering and sensitivity remain areas needing attention.
  • Cross-lingual Transfer Gaps: While top proprietary models excel in English, their performance can lag in Chinese, suggesting room for improvement in cross-lingual capabilities.
  • Data Watermarking Trade-offs: While effective for deterring attackers, data watermarking can negatively impact model performance, necessitating careful application.

Key Results

  • C²LEVA was used to evaluate 15 LLMs (both open-source and proprietary).
  • Proprietary models generally outperformed open-source ones, and larger models tended to be stronger.
  • Claude-3.5, Gemini-1.5, and GPT-40 emerged as top performers, particularly in English.
  • The C²LEVA benchmark showed a strong linear correlation (Spearman's ρ = 0.948) with the Chatbot Arena Elo ranking, suggesting it is comprehensive and effectively mitigates data contamination.
  • Data watermarking led to performance declines in 76.67% of cases, with an average loss of ~11.59%, especially noticeable in Chinese.

Limitations and Future Work

  • Task Incompleteness: C²LEVA is in its early stages and doesn't include all possible NLP tasks. Future work will incorporate more tasks.
  • Single Metric Focus: The current evaluation primarily uses accuracy. Extending to multi-metric evaluation is planned.
  • Simple Active Prevention: The active prevention method is relatively simple and may cause evaluation distortion. More sophisticated algorithmic designs are anticipated.
  • Cross-lingual Improvement: Further research is needed to enhance cross-lingual transfer capabilities.
  • Data Watermarking Refinement: Developing data protection techniques with less performance impact is a key area for future work.

Summary updated

Abstract

Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C2LEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. C2LEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of C2LEVA.
BibTeX
@inproceedings{li2025c,
  author = {Yanyang Li and Tin Long Wong and Cheung To Hung and Jianqiao Zhao and Duo Zheng and Ka Wai Liu and Michael R. Lyu and Liwei Wang},
  title = {{C²LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure