ArcMMLU: A Library and Information Science Benchmark for Large Language Models

Shitou Zhang, Zuchao Li, Xingshen Liu, Liming Yang, Ping WangarXiv 2023 · DOI · arXiv

Introduction

As Large Language Models (LLMs) continue to advance, standard benchmarks like MMLU or GLUE are increasingly insufficient for distinguishing between top-tier models, particularly in specialized professional domains. One such critical but overlooked field is Library and Information Science (LIS). LIS encompasses archival management, complex information retrieval, and data organization—tasks that require specific disciplinary knowledge and logic.

ArcMMLU is a new, specialized benchmark designed to evaluate LLMs' proficiency in the Chinese LIS context. By providing a comprehensive assessment across four sub-domains, ArcMMLU aims to fill the evaluation void and guide the development of AI assistants tailored for librarians and archivists.

Methodology

The construction of ArcMMLU followed a rigorous six-stage workflow to ensure high data quality and professional relevance.

The six-stage construction workflow of the ArcMMLU benchmark.
  1. Raw Data Collection: Questions were sourced from real-world postgraduate entrance exams, professional certifications, and academic competitions.
  2. Pre-processing: Text was extracted from various formats using OCR and PDF parsing.
  3. Filtering: Non-textual elements (images/tables) were removed, and entries were deduplicated.
  4. Data Annotation: LIS graduate students and professors manually labeled and cross-inspected the data to resolve inconsistencies.
  5. Quality Checking: Random sampling and manual inspection ensured accuracy.
  6. Post-processing: Data was formatted into a standard four-option single-choice structure compatible with MMLU/CMMLU formats.

Dataset Characteristics

The resulting ArcMMLU dataset consists of 6,210 high-quality questions distributed across four primary sub-domains:

  • Archival Science: 2,213 questions
  • Information Science: 1,674 questions
  • Data Science: 1,499 questions
  • Library Science: 804 questions

The dataset includes a development set of 20 questions for few-shot prompting and a test set of 6,190 questions. Additionally, a subset called ArcMMLU-Hard was created, containing questions that all evaluated open-source models failed to answer, serving as a benchmark for the absolute reasoning ceiling of current LLMs.

Results and Analysis

The evaluation involved 14 model families tested in zero-shot and few-shot (up to 5-shot) settings.

Performance of various LLMs across four LIS sub-domains under zero-shot and few-shot settings.

Model Performance

GPT-4 emerged as the dominant performer, achieving an average accuracy of 75.08% in the 5-shot setting. Among open-source models, Alibaba’s Qwen-14B led with 69.07%. While most mainstream LLMs scored above the 25% random-guess baseline, there remains significant room for improvement to reach professional-level proficiency.

The "Few-Shot Paradox"

A notable finding was the performance decline in certain models, such as ChatGLM-6B, when provided with few-shot examples. This "Few-Shot Paradox" suggests either weak in-context generalization or a disruption of the model's pre-trained knowledge by the provided context.

Domain Variance

Models generally performed better in Data Science and Library Science, likely due to their overlap with Computer Science and Management—fields well-represented in general training corpora. Conversely, Archival Science and Information Science proved more challenging, highlighting their specialized and self-contained nature.

Conclusion

ArcMMLU provides a vital tool for assessing LLMs in the LIS domain. The results underscore the need for "domain-injected" training to bridge the performance gap in highly specialized fields like archival science. Future work will expand the benchmark to include more sub-domains and multi-lingual data, further refining the evaluation of AI in professional library and information contexts.

Summary updated

Abstract

In light of the rapidly evolving capabilities of large language models (LLMs), it becomes imperative to develop rigorous domain-specific evaluation benchmarks to accurately assess their capabilities. In response to this need, this paper introduces ArcMMLU, a specialized benchmark tailored for the Library & Information Science (LIS) domain in Chinese. This benchmark aims to measure the knowledge and reasoning capability of LLMs within four key sub-domains: Archival Science, Data Science, Library Science, and Information Science. Following the format of MMLU/CMMLU, we collected over 6,000 high-quality questions for the compilation of ArcMMLU. This extensive compilation can reflect the diverse nature of the LIS domain and offer a robust foundation for LLM evaluation. Our comprehensive evaluation reveals that while most mainstream LLMs achieve an average accuracy rate above 50% on ArcMMLU, there remains a notable performance gap, suggesting substantial headroom for refinement in LLM capabilities within the LIS domain. Further analysis explores the effectiveness of few-shot examples on model performance and highlights challenging questions where models consistently underperform, providing valuable insights for targeted improvements. ArcMMLU fills a critical gap in LLM evaluations within the Chinese LIS domain and paves the way for future development of LLMs tailored to this specialized area.
BibTeX
@misc{zhang2023arcmmlu,
  author = {Shitou Zhang and Zuchao Li and Xingshen Liu and Liming Yang and Ping Wang},
  title = {{ArcMMLU: A Library and Information Science Benchmark for Large Language Models}},
  year = {2023},
  eprint = {2311.18658},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.18658},
}
Figure