Abstract
Cognitive Diagnosis Models (CDMs) are designed to assess students' cognitive states by analyzing their performance across a series of exercises. However, existing CDMs often struggle with diagnosing infrequent students and exercises due to a lack of rich prior knowledge. With the advancement in large language models (LLMs), which possess extensive domain knowledge, their integration into cognitive diagnosis presents a promising opportunity. Despite this potential, integrating LLMs with CDMs poses significant challenges. LLMs are not well-suited for capturing the fine-grained collaborative interactions between students and exercises, and the disparity between the semantic space of LLMs and the behavioral space of CDMs hinders effective integration. To address these issues, we propose a novel Knowledge-enhanced Cognitive Diagnosis (KCD) framework, which is a model-agnostic framework utilizing LLMs to enhance CDMs and compatible with various CDM architectures. The KCD framework operates in two stages: LLM Diagnosis and Cognitive Level Alignment. In the LLM Diagnosis stage, both students and exercises are diagnosed to achieve comprehensive and detailed modeling. In the Cognitive Level Alignment stage, we bridge the gap between the CDMs' behavioral space and the LLMs' semantic space using contrastive learning and mask-reconstruction approaches. Experiments on several real-world datasets demonstrate the effectiveness of our proposed framework.
Introduction
Cognitive Diagnosis Models (CDMs) serve as the backbone of intelligent tutoring systems by estimating a student's proficiency across various knowledge concepts based on their exercise history. However, traditional CDMs face two major hurdles: the cold-start problem and a lack of contextual understanding. When a student or exercise is new to the system, data-driven models lack sufficient interaction history to make accurate diagnoses. Furthermore, most CDMs treat exercises as simple IDs, ignoring the rich textual information and domain knowledge inherent in the questions themselves.
The KCD Framework
To address these limitations, we introduce the Knowledge-enhanced Cognitive Diagnosis (KCD) framework. KCD is a model-agnostic approach that leverages the vast prior knowledge of Large Language Models (LLMs) to augment traditional CDMs. The framework operates in two distinct stages:
1. LLM Diagnosis
In this stage, an LLM (specifically GPT-3.5-turbo) is prompted to act as an "experienced teacher." It performs collaborative information collection by summarizing a student's learning status based on their history and describing the specific traits of exercises. This results in textual descriptions of a student's cognitive state and the attributes required for an exercise.
2. Cognitive Level Alignment
A core challenge is bridging the gap between the LLM's semantic space (textual meaning) and the CDM's behavioral space (interaction patterns). KCD employs two alignment strategies:
- Behavioral Space Alignment (KCD-Beh): Uses contrastive learning to map LLM-generated textual features into the CDM’s behavioral space, utilizing both global and local contrast to refine the diagnosis.
- Semantic Space Alignment (KCD-Sem): Maps CDM embeddings back into the LLM’s semantic space using a dynamic masking strategy. This strategy adjusts the learning difficulty based on data frequency, forcing the model to learn deeper features for frequent students while remaining robust for cold-start scenarios.
Experimental Evaluation
The KCD framework was evaluated using the PTADisc dataset, which contains real-world student responses across four domains: Python Programming, Linux Systems, Database Technology, and Literature and History. These datasets vary in sparsity, with the Python dataset being particularly sparse (0.063%), providing a rigorous test for cold-start performance.
Key Results and Discussion
KCD consistently outperformed seven baseline models (including IRT, NCD, and RCD) across metrics like AUC, Accuracy (ACC), and RMSE. For instance, on the Python dataset, KCD-Beh improved the AUC of the Neural Cognitive Diagnosis (NCD) model from 0.6522 to 0.6804.
The framework's strength in handling the cold-start problem is particularly evident. In scenarios where students or exercises have very few interactions, the LLM's prior knowledge successfully compensates for the lack of behavioral data, leading to more accurate and stable diagnoses compared to traditional methods.
Conclusion
The KCD framework demonstrates that integrating LLMs into cognitive diagnosis can effectively bridge the gap between semantic knowledge and behavioral data. By aligning these two spaces, KCD not only improves overall diagnostic accuracy but also provides a powerful solution to the long-standing cold-start problem in intelligent tutoring systems. Future work will explore incorporating student affective states and more complex graph-based relationships between knowledge concepts.
Summary updated