Automated Generation and Tagging of Knowledge Components from Multiple-Choice Questions

Steven Moore, Robin Schmucker, Tom Mitchell, John Stamper

This paper presents a novel approach to a long-standing bottleneck in educational technology: the manual creation of Knowledge Components (KCs). By leveraging GPT-4, the researchers demonstrate that AI can not only generate these fine-grained skill labels for assessment items but often does a better job than the human experts who originally designed the courses.

TL;DR

Researchers used GPT-4 to automatically generate and tag Knowledge Components for college-level questions, finding that domain experts preferred AI-generated labels over human-created ones roughly two-thirds of the time. They also introduced an algorithm that can group related questions into a structured "ontology" without needing any prior labels or student data.

What Problem Does This Paper Address?

In digital learning, a Knowledge Component (KC) is a specific mental representation or skill required to solve a problem (e.g., "applying the concept of electronegativity"). A collection of these KCs forms a Knowledge Component Model (KCM), which acts as the "brain" behind adaptive learning systems, allowing them to track exactly what a student knows and what they need to practice next.

The challenge? Building these models is incredibly labor-intensive. It requires domain experts to painstakingly map every single question to specific skills. This process is slow, expensive, and prone to "expert blind spots"—where a teacher is so familiar with a topic they overlook the granular sub-skills a novice needs to learn.

Methods

The authors tested two primary prompting strategies using the GPT-4 API:

  1. Simulated Expert: Uses a "Tree-of-Thought" approach where the LLM emulates a discussion between three "brilliant, nit-picky" instructors to reach a consensus on the skills assessed by a question.
  2. Simulated Textbook: Contextualizes the question as if it were appearing in a textbook, asking the LLM what specific low-level topics the surrounding page would cover.

To ensure the labels were pedagogically sound, they used a follow-up prompt to align the generated KCs with Bloom’s Revised Taxonomy (using action verbs like "analyze" or "evaluate").

Finally, they developed a KC Ontology Induction algorithm. This is an iterative clustering process that asks the LLM to partition a pool of questions into groups based on shared learning objectives, effectively building a hierarchy of skills from the bottom up.

Figure 3: A Chemistry MCQ (top) and E-Learning MCQ (bottom) used in this study.

Datasets

The study utilized two distinct datasets from higher-education MOOCs:

  • Chemistry: 80 multiple-choice questions (MCQs) from an introductory undergraduate course.
  • E-Learning: 80 MCQs from a master’s level course on instructional design. Each dataset was structured so that every KC was represented by exactly two questions, providing a "gold standard" for evaluating how well the LLM could identify shared concepts.

Key Insights and Ideas

  • The "Gold Standard" is Moving: The most striking insight is that "ground truth" human labels aren't always the best. When the LLM's tag didn't match the human's tag, experts were asked to pick the better one in a blind test. They chose the LLM's version ~66% of the time.
  • Domain Sensitivity: The LLM performed significantly better in Chemistry (56% direct match) than in E-Learning (35%). This suggests that "hard" sciences with more standardized jargon may be easier for current models to parse than "soft" sciences with more fluid terminology.
  • Zero-Data Cold Start: Unlike traditional methods that require thousands of student performance records to "discover" KCs statistically, this method works using only the text of the questions themselves. This is a game-changer for new courses that haven't been deployed yet.

Key Results

  • Preference: In a statistically significant result, human evaluators favored LLM-generated KCs over human-assigned ones in both domains. A bar chart showing that experts consistently preferred LLM labels over the original human labels.

  • Accuracy: While direct matches were moderate, the LLM's "Top 5" suggestions contained the correct human-assigned KC 79-80% of the time for Chemistry.

  • Clustering: The ontology algorithm successfully converged to a model of 42 KCs for Chemistry (very close to the expert model of 40), showing that AI can effectively organize a messy pile of questions into a structured curriculum.
Figure 6: Comparison of domain expert preferences for human- vs. LLM-generated KC labels.
Figure 7: A section of the tree structure demonstrating KC ontology refinement for part of Chemistry.

Limitations and Future Work

The authors acknowledge that LLMs can be sensitive to prompt wording and may struggle with highly niche domains that weren't well-represented in their training data. Furthermore, the study was limited to multiple-choice questions; future work will need to explore if these results hold for open-ended or short-answer questions.

The researchers suggest a "human-in-the-loop" workflow: let the LLM do the heavy lifting of generating the initial KCM, and then have a domain expert perform a final verification pass. This could reduce the time required to build adaptive courseware from months to days.

Summary updated

Abstract

Knowledge Components (KCs) linked to assessments enhance the measurement of student learning, enrich analytics, and facilitate adaptivity. However, generating and linking KCs to assessment items requires significant effort and domain-specific knowledge. To streamline this process for higher-education courses, we employed GPT-4 to generate KCs for multiple-choice questions (MCQs) in Chemistry and E-Learning. We analyzed discrepancies between the KCs generated by the Large Language Model (LLM) and those made by humans through evaluation from three domain experts in each subject area. This evaluation aimed to determine whether, in instances of non-matching KCs, evaluators showed a preference for the LLM-generated KCs over their human-created counterparts. We also developed an ontology induction algorithm to cluster questions that assess similar KCs based on their content. Our most effective LLM strategy accurately matched KCs for 56% of Chemistry and 35% of E-Learning MCQs, with even higher success when considering the top five KC suggestions. Human evaluators favored LLM-generated KCs, choosing them over human-assigned ones approximately two-thirds of the time, a preference that was statistically significant across both domains. Our clustering algorithm successfully grouped questions by their underlying KCs without needing explicit labels or contextual information. This research advances the automation of KC generation and classification for assessment items, alleviating the need for student data or predefined KC labels.
BibTeX
@inproceedings{moore2024automated,
  author = {Steven Moore and Robin Schmucker and Tom Mitchell and John Stamper},
  title = {{Automated Generation and Tagging of Knowledge Components from Multiple-Choice Questions}},
  booktitle = {ACM Conference on Learning @ Scale},
  year = {2024},
  url = {https://openreview.net/forum?id=EFfqgi2TC7},
}
Figure