Introduction
As Large Language Models (LLMs) continue to advance, standard benchmarks like MMLU or GLUE are increasingly insufficient for distinguishing between top-tier models, particularly in specialized professional domains. One such critical but overlooked field is Library and Information Science (LIS). LIS encompasses archival management, complex information retrieval, and data organization—tasks that require specific disciplinary knowledge and logic.
ArcMMLU is a new, specialized benchmark designed to evaluate LLMs' proficiency in the Chinese LIS context. By providing a comprehensive assessment across four sub-domains, ArcMMLU aims to fill the evaluation void and guide the development of AI assistants tailored for librarians and archivists.
Methodology
The construction of ArcMMLU followed a rigorous six-stage workflow to ensure high data quality and professional relevance.
- Raw Data Collection: Questions were sourced from real-world postgraduate entrance exams, professional certifications, and academic competitions.
- Pre-processing: Text was extracted from various formats using OCR and PDF parsing.
- Filtering: Non-textual elements (images/tables) were removed, and entries were deduplicated.
- Data Annotation: LIS graduate students and professors manually labeled and cross-inspected the data to resolve inconsistencies.
- Quality Checking: Random sampling and manual inspection ensured accuracy.
- Post-processing: Data was formatted into a standard four-option single-choice structure compatible with MMLU/CMMLU formats.
Dataset Characteristics
The resulting ArcMMLU dataset consists of 6,210 high-quality questions distributed across four primary sub-domains:
- Archival Science: 2,213 questions
- Information Science: 1,674 questions
- Data Science: 1,499 questions
- Library Science: 804 questions
The dataset includes a development set of 20 questions for few-shot prompting and a test set of 6,190 questions. Additionally, a subset called ArcMMLU-Hard was created, containing questions that all evaluated open-source models failed to answer, serving as a benchmark for the absolute reasoning ceiling of current LLMs.
Results and Analysis
The evaluation involved 14 model families tested in zero-shot and few-shot (up to 5-shot) settings.
Model Performance
GPT-4 emerged as the dominant performer, achieving an average accuracy of 75.08% in the 5-shot setting. Among open-source models, Alibaba’s Qwen-14B led with 69.07%. While most mainstream LLMs scored above the 25% random-guess baseline, there remains significant room for improvement to reach professional-level proficiency.
The "Few-Shot Paradox"
A notable finding was the performance decline in certain models, such as ChatGLM-6B, when provided with few-shot examples. This "Few-Shot Paradox" suggests either weak in-context generalization or a disruption of the model's pre-trained knowledge by the provided context.
Domain Variance
Models generally performed better in Data Science and Library Science, likely due to their overlap with Computer Science and Management—fields well-represented in general training corpora. Conversely, Archival Science and Information Science proved more challenging, highlighting their specialized and self-contained nature.
Conclusion
ArcMMLU provides a vital tool for assessing LLMs in the LIS domain. The results underscore the need for "domain-injected" training to bridge the performance gap in highly specialized fields like archival science. Future work will expand the benchmark to include more sub-domains and multi-lingual data, further refining the evaluation of AI in professional library and information contexts.
Summary updated