CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation

Weihao You, Pengcheng Wang, Changlong Li, Zhilong Ji, Jinfeng BaiAAAI 2024 · DOI

Abstract

Existing benchmarks for Large Language Models (LLMs) often suffer from "ceiling effects" or rely on limited examination papers that fail to cover the full breadth of educational curricula. We introduce CK12, a comprehensive benchmark for evaluating LLMs within the Chinese K12 (primary through high school) educational context. CK12 is built upon a multi-level knowledge graph containing nearly 40,000 questions across nine subjects. By incorporating robust testing mechanisms such as distractor options and process-oriented reasoning evaluations, CK12 provides a more nuanced and "holistic" assessment of a model's cognitive and logical capabilities compared to traditional result-oriented benchmarks.

Introduction

As LLMs continue to advance, standard benchmarks like MMLU or C-Eval are increasingly hitting performance ceilings, making it difficult to distinguish between top-tier models. Furthermore, many Chinese benchmarks are constructed from past exam papers, which are inherently limited in scope and may leave significant "holes" in knowledge evaluation. These benchmarks also frequently focus solely on final answers, allowing models to potentially "cheat" through data contamination or lucky guesses. CK12 addresses these issues by shifting the focus toward a comprehensive, knowledge-point-driven evaluation that tests both factual retrieval and logical reasoning.

The CK12 Knowledge Graph and Methodology

The core of CK12 is a hierarchical Knowledge Graph (KG) that structures educational content into three distinct levels: Subject, Level-1 points, and Level-2 points. This structure ensures that every niche area of the curriculum is represented.

The CK12 Knowledge Graph structure, illustrating the hierarchy from subjects to specific knowledge points.

To ensure robustness, the researchers implemented several evaluation strategies:

  • Prompting Modes: Models are tested using Zero-shot, Few-shot, and Chain-of-Thought (CoT) prompts.
  • Robustness Testing: To combat memorization, the benchmark utilizes option shuffling and "Interference Options"—adding ten distractor choices from the same subject to test true conceptual understanding.
  • Process-Oriented Evaluation: For mathematics, the benchmark evaluates the reasoning steps using metrics like "Reasoning Alignment" to ensure the model's logic matches human-like problem-solving.

Dataset Composition

The CK12 dataset comprises 39,452 questions covering nine core subjects: Chinese, Math, English, Physics, Chemistry, Biology, History, Politics, and Geography. It spans 584 Level-1 and 1,989 Level-2 knowledge points.

Distribution of subjects within the CK12 dataset and a detailed breakdown of math knowledge points.

A significant portion of the dataset (7,334 questions) is dedicated to a "Reasoning Subset" of math problems requiring multiple logical steps, sourced from private test banks and verified by professional educators.

Experimental Results and Insights

Evaluation of various LLMs on CK12 revealed several key trends:

  • The Arts vs. Science Gap: Models generally perform better in "Arts" subjects (History, Geography) than in "Sciences" (Physics, Math), often by a margin of over 10 percentage points. This suggests that while retrieval is strong, deep logical derivation remains a challenge.
  • Sensitivity to Distractors: The introduction of interference options led to a significant drop in performance across all models, highlighting the "brittleness" of current LLMs when faced with increased noise.
  • Specialization over Scale: Smaller, Chinese-specific models like InternLM-7B frequently outperformed larger general models like Llama-65B, emphasizing the value of high-quality, domain-specific training data.
  • CoT Performance: Interestingly, some instruction-tuned models showed degraded performance when using Chain-of-Thought prompts, likely due to overfitting on specific instruction formats.

Conclusion

CK12 provides a rigorous and holistic framework for evaluating the cognitive depth of LLMs in the Chinese K12 domain. While GPT-4 currently sets the performance benchmark, the study identifies clear "blind spots" in many models, such as specific failures in Atomic Physics or Science Practice. Future work will focus on expanding the benchmark to university-level tasks and releasing high-quality explanations to aid the open-source community in improving model reasoning.

Summary updated

BibTeX
@inproceedings{you2024ck,
  author = {Weihao You and Pengcheng Wang and Changlong Li and Zhilong Ji and Jinfeng Bai},
  title = {{CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29914},
}
Figure