Abstract
Cognitive Diagnosis Models (CDMs) are designed to assess students' cognitive states by analyzing their performance across a series of exercises. However, existing CDMs often struggle with diagnosing infrequent students and exercises due to a lack of rich prior knowledge. With the advancement in large language models (LLMs), which possess extensive domain knowledge, their integration into cognitive diagnosis presents a promising opportunity. Despite this potential, integrating LLMs with CDMs poses significant challenges. LLMs are not well-suited for capturing the fine-grained collaborative interactions between students and exercises, and the disparity between the semantic space of LLMs and the behavioral space of CDMs hinders effective integration. To address these issues, we propose a novel Knowledge-enhanced Cognitive Diagnosis (KCD) framework, which is a model-agnostic framework utilizing LLMs to enhance CDMs and compatible with various CDM architectures. The KCD framework operates in two stages: LLM Diagnosis and Cognitive Level Alignment. In the LLM Diagnosis stage, both students and exercises are diagnosed to achieve comprehensive and detailed modeling. In the Cognitive Level Alignment stage, we bridge the gap between the CDMs' behavioral space and the LLMs' semantic space using contrastive learning and mask-reconstruction approaches. Experiments on several real-world datasets demonstrate the effectiveness of our proposed framework.
Introduction
Cognitive Diagnosis (CD) is a cornerstone of personalized education, enabling the assessment of specific concept mastery based on student performance. Traditional Cognitive Diagnosis Models (CDMs) rely heavily on interaction logs. However, they face a significant cold-start problem: when a student is new or an exercise is rare, the lack of data leads to inaccurate diagnoses.
Large Language Models (LLMs) offer a potential solution through their vast domain knowledge. However, LLMs operate in a semantic space (textual meaning), while CDMs operate in a behavioral space (interaction patterns). Bridging this gap is essential for effective integration. We introduce the Knowledge-enhanced Cognitive Diagnosis (KCD) framework to address these challenges by injecting LLM-derived prior knowledge into traditional CDMs.
Methods
The KCD framework is model-agnostic and consists of two primary stages:
- LLM Diagnosis: The system prompts an LLM to act as an "experienced teacher," processing collaborative summaries of student performance and exercise traits. The LLM generates textual diagnoses that describe a student's cognitive state and the attributes of an exercise.
- Cognitive Level Alignment: This stage aligns the LLM's semantic insights with the CDM's behavioral data using two strategies:
- Behavioral Space Alignment: Uses contrastive learning (InfoNCE loss) to ensure CDM representations are mathematically close to LLM diagnoses.
- Semantic Space Alignment: Employs a mask-reconstruction approach (inspired by Masked Autoencoders) to reconstruct behavioral features from semantic features, using a dynamic masking strategy to force deeper learning for frequent students.
Results and Discussion
Experiments on the PTADisc dataset (covering Python, Linux, Database, and Literature) demonstrate that KCD consistently improves the performance of various baseline models.
Cold-Start Performance
KCD significantly mitigates the cold-start problem. For students with fewer than three interactions, the KCD variants (NCD-Beh and NCD-Sem) show substantial improvements in AUC and Accuracy compared to the standard NCD model. This confirms that LLM-derived semantic information effectively compensates for sparse behavioral data.
Alignment Visualization
Using t-SNE plots, we observed that after alignment, the mathematical clusters of students in the CDM's behavioral space and the LLM's semantic space began to overlap. This visual evidence confirms that the alignment strategies successfully bridged the gap between interaction patterns and textual descriptions.
Conclusion
The KCD framework demonstrates the power of leveraging LLMs as prior knowledge sources for cognitive diagnosis. By aligning semantic and behavioral spaces, it provides a robust solution to the cold-start problem. Future work will explore more complex alignment methods and the use of specialized, fine-tuned educational LLMs to optimize context handling and computational efficiency.
Summary updated