Evalverse: Unified and Accessible Library for Large Language Model Evaluation

Jihoo Kim, Wonho Song, Dahyun Kim, Yunsu Kim, Yungi Kim, Chanjun ParkEMNLP 2024

TL;DR

Evalverse is a new Python library that aims to simplify and unify the evaluation of Large Language Models (LLMs). It integrates various existing evaluation frameworks into a single, expandable library and offers a no-code interface via Slack, making LLM evaluation more accessible to a wider audience.

What Problem Does This Paper Address?

The rapid advancement of LLMs has led to a proliferation of evaluation tools and methodologies. This fragmentation makes it difficult for researchers and practitioners to compare LLMs effectively, creating a significant barrier to entry. The paper addresses this by proposing Evalverse, a unified and accessible framework that streamlines LLM evaluation.

Methods

Evalverse's core innovation lies in its architecture, which comprises six key components:

  • Submodule: Acts as the evaluation engine, allowing integration of publicly available LLM evaluation libraries (e.g., Im-evaluation-harness, FastChat) as submodules. This makes Evalverse expandable and ensures it can stay up-to-date with new benchmarks.
  • Connector: Links submodules with the Evaluator, managing evaluation scripts and arguments.
  • Evaluator: Executes the requested evaluations on a Compute Cluster. It can receive requests directly from users (for code-based evaluation) or via the Reporter (for no-code evaluation).
  • Compute Cluster: Provides the necessary hardware accelerators for running evaluations.
  • Database: Stores model files, data, and evaluation results.
  • Reporter: Enables a no-code approach by handling user requests (via Slack) for evaluations and reports. It sends requests to the Evaluator and retrieves results from the Database.

A key feature is the no-code evaluation via Slack. Users can simply type "Request!" in a Slack channel, and the Evalverse bot guides them through selecting models and evaluation criteria. Similarly, users can request reports by typing "Report!".

The library also emphasizes unified and expandable evaluation by integrating existing frameworks as submodules. This allows for easy addition of new benchmarks and fetching of upstream changes to stay current.

Datasets

The paper does not introduce new benchmarks. Instead, Evalverse integrates and evaluates LLMs on existing benchmarks such as:

  • General Performance: H6 Avg (combining ARC-Challenge, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8k)
  • Chat Applications: MT-Bench, IFEval, EQ-Bench
  • Retrieval Augmented Generation (RAG): RGB, Format-Following benchmark (FoFo)
  • Domain-Specific: FinGPT Benchmark, MultiMedQA, Legal-Bench

Key Insights and Ideas

  • Democratizing LLM Evaluation: The no-code interface via Slack is a significant step towards making LLM evaluation accessible to individuals without deep programming expertise.
  • Unified Ecosystem: By integrating disparate evaluation tools into a single framework, Evalverse reduces the friction of comparing LLMs across different benchmarks.
  • Extensibility is Key: The submodule architecture is crucial for keeping Evalverse relevant in the rapidly evolving LLM landscape. It allows for easy incorporation of new evaluation frameworks and benchmarks.
  • Reproducibility Matters: The paper highlights the importance of reproducible evaluation results, showing that Evalverse can largely reproduce scores from original repositories, with minor differences attributed to specific prompt versions or intentional modifications.

Key Results

  • Reproducibility: Table 3 shows that Evalverse produces similar evaluation scores to original repositories for benchmarks with minimal modifications (H6, MT-Bench). Differences in EQ-Bench are attributed to prompt variations.
  • Evaluation Speed: Table 4 indicates that Evalverse can be faster for some benchmarks (e.g., EQ-Bench) due to added data parallelism support.
  • Comprehensive Evaluation: Table 5 demonstrates Evalverse's capability to evaluate various open-source models across multiple benchmarks, providing insights into their performance for pre-trained and fine-tuned tasks.

Limitations and Future Work

  • Maintenance Overhead: Keeping Evalverse up-to-date with the rapidly evolving LLM evaluation landscape requires ongoing commitment.
  • Interpretation Complexity: While Evalverse makes evaluation accessible, interpreting the results may still require specialized knowledge.
  • Third-Party Dependencies: Reliance on platforms like Slack introduces dependencies that could affect long-term sustainability and adaptability.
  • Future Work: The authors mention expanding to other communication tools as an important milestone and aim to foster a more inclusive community by providing accessible tools and encouraging contributions.

Summary updated

BibTeX
@inproceedings{kim2024evalverse,
  author = {Jihoo Kim and Wonho Song and Dahyun Kim and Yunsu Kim and Yungi Kim and Chanjun Park},
  title = {{Evalverse: Unified and Accessible Library for Large Language Model Evaluation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure