EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMs

Numaan Naeem, Abdellah El Mekki, Muhammad Abdul-MageedEMNLP 2025

TL;DR

Large Language Models (LLMs) excel at answering questions but often fail to adapt their explanations to different grade levels, posing a challenge for K-12 education. This paper introduces EDUADAPT, a novel benchmark of nearly 48,000 grade-labeled QA pairs across nine science subjects, designed to evaluate LLMs' ability to tailor responses to students from Grades 1-12. The findings reveal a significant performance gap, with current LLMs struggling to generate age-appropriate content for younger learners, highlighting the need for more developmentally aligned AI systems.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has opened up exciting possibilities for educational applications, from answering complex questions to generating explanatory content. However, a critical limitation has emerged: LLMs often struggle to adapt their output to the specific grade level of the student. This is particularly problematic in K-12 education, where vocabulary, conceptual understanding, and cognitive abilities vary significantly across age groups. Existing LLMs frequently produce responses that are either too advanced or too simplistic for younger learners, and there's a lack of standardized benchmarks to systematically evaluate this "grade-level adaptability." This gap hinders the development of truly effective and equitable AI-powered educational tools that can cater to the diverse needs of students across the entire K-12 spectrum.

Methods

The creation of the EDUADAPT benchmark involved a two-stage process:

  1. Generation Process:
    • Content Extraction: Clean, domain-specific text was extracted from Wikipedia articles across nine subjects (Chemistry, Computer Science, Meteorology, Ecology, Geology, Biology, Physics, Medicine, and Geography).
  2. QA Generation: A reliable LLM was used to generate initial question-answer (QA) pairs tailored to different grade levels (Grades 1-12, grouped into four levels: 1-2, 3-5, 6-8, 9-12). Both multiple-choice questions (MCQs) and open-ended QA were generated.
  3. Self-Reflection: An LLM was employed to critique its own generated QA pairs based on pedagogical and linguistic criteria (language appropriateness, grade alignment, relevance, clarity, subject fit). This iterative refinement process helped filter out lower-quality pairs.
  4. Filtering: Only QA pairs scoring at least 8 out of 10 on all self-reflection criteria were retained.
  5. Human Verification Process:
    • A subset of the generated QA pairs (approximately 10% of the test set) was manually reviewed by three human annotators with backgrounds in educational content development.
  6. Reviewers assessed the QA pairs based on the same criteria used in the self-reflection stage to ensure quality and grade-level appropriateness.

Datasets

The paper introduces EDUADAPT, a new benchmark dataset comprising approximately 48,000 grade-labeled QA pairs. This dataset spans nine science subjects and covers Grades 1-12, organized into four grade-level bands: Grades 1-2, 3-5, 6-8, and 9-12. The QA pairs are aligned with the Next Generation Science Standards (NGSS) to ensure coverage from basic recall to higher-order reasoning. The dataset is divided into training, development, and testing sets.

Key Insights and Ideas

  • The "Grade Gap" is Real: The most striking insight is the consistent and significant performance disparity between LLMs on early-grade content (Grades 1-5) versus higher-grade content (Grades 6-12). This isn't just a minor issue; it points to a fundamental challenge in LLM development for educational applications.
  • Surface-Level Metrics Fall Short: Traditional NLP metrics like BLEU, ROUGE, and even BERTScore proved unreliable for evaluating educational QA. They often captured surface-level similarity but missed crucial aspects like factual accuracy, conceptual alignment, and developmental appropriateness. This underscores the need for more nuanced evaluation methods.
  • LLM-as-a-Judge is Promising, but Needs Care: The LLM-as-a-judge framework, when carefully prompted with specific criteria, offers a scalable way to evaluate open-ended responses. However, the analysis of errors revealed that even LLM judges can struggle with nuanced aspects of grade-level appropriateness, highlighting the continued importance of human oversight.
  • Smaller Models Struggle More: While larger models generally perform better, smaller models (1.5B-3B parameters) exhibit particularly poor performance on lower grade levels, often falling below 60% accuracy on open-ended questions. This suggests that simply scaling up might not be enough to achieve true grade-level adaptability.

Key Results

  • MCQ Performance: Larger models like Qwen2.5-14B and Mistral-Small-24B achieved the highest accuracy on MCQs, reaching up to 86%. However, even these models struggled with lower grades (1-5), with accuracy dropping to around 50-60% for smaller models.
  • Open-Ended Response Quality: For open-ended questions, evaluated using an LLM-as-a-judge framework, all models showed a clear trend of better performance on higher grades. Larger models like Qwen2.5-14B and Mistral-Small-24B consistently scored higher, but still exhibited weaker performance on early-grade content.
  • Error Analysis: Qualitative analysis revealed recurring issues across models, including hallucination, focus drift, correctness errors, and conceptual alignment problems, particularly for younger learners.
  • Human Evaluation: Human evaluation of a test set sample showed strong inter-rater agreement, reinforcing the dataset's reliability. Average scores for language appropriateness and grade alignment were consistently lower for earlier grades.

Limitations and Future Work

The authors acknowledge several limitations:

  • Grade-Level Imbalance: The dataset has fewer QA pairs for lower grades (1-5) compared to upper grades (6-12). Future work should aim to create more balanced datasets.
  • K-12 Curriculum Framework: The dataset is based on a K-12 framework, which may limit its generalizability to other educational systems with different curricular standards.
  • Regional Variations: While subjects like ecology and geography draw examples from diverse regions, the benchmark doesn't account for regional variations in curricular relevance.

Future work directions include:

  • Expanding Subject Coverage: Incorporating more subjects to achieve broader curriculum alignment.
  • Multimodal QA: Developing QA pairs that include images or diagrams to better reflect real-world assessments.
  • Multilingual QA: Increasing accessibility for non-English-speaking students.
  • Data Augmentation and Pretraining: Addressing lower-grade performance through data augmentation, curriculum-aligned pretraining, and targeted fine-tuning strategies.
  • Culturally and Regionally Grounded Evaluation: Refining the benchmark to account for cultural and regional differences in educational contexts.

Summary updated

BibTeX
@inproceedings{naeem2025eduadapt,
  author = {Numaan Naeem and Abdellah El Mekki and Muhammad Abdul-Mageed},
  title = {{EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure