Enhancing Length Generalization for Attention Based Knowledge Tracing Models with Linear Biases

Xueyi Li, Youheng Bai, Teng Guo, Zitao Liu, Yaying Huang, Xiangyu Zhao, Feng Xia, Weiqi Luo, Jian Weng

Abstract

Knowledge tracing (KT) is the task of predicting students' future performance based on their historical learning interaction data. With the rapid advancement of attention mechanisms, many attention based KT models are developed. However, existing attention based KT models exhibit performance drops as the number of student interactions increases beyond the number of interactions on which the KT models are trained. We refer to this as the length generalization of KT model. In this paper, we propose stableKT to enhance length generalization that is able to learn from short sequences and maintain high prediction performance when generalizing on long sequences. Furthermore, we design a multi-head aggregation module to capture the complex relationships between questions and the corresponding knowledge components (KCs) by combining dot-product attention and hyperbolic attention. Experimental results on three public educational datasets show that our model exhibits robust capability of length generalization and outperforms all baseline models in terms of AUC.

Introduction

Knowledge Tracing (KT) serves as the foundational engine for personalized learning systems, estimating a student's mastery of specific skills (Knowledge Components or KCs) to predict future performance. While Transformer-based models have achieved state-of-the-art results in KT, they face a significant challenge: "length generalization."

Existing models often struggle when student interaction histories grow longer than the sequences seen during training. For instance, a model trained on 200 interactions may see its accuracy plummet when evaluating a student with 1,000 interactions. This occurs because standard position embeddings do not extrapolate well to unseen lengths. Additionally, traditional models often ignore the hierarchical nature of educational concepts (e.g., the prerequisite relationship between "Numbers" and "Negative Numbers"), treating all KCs as independent entities in Euclidean space.

Methods: The stableKT Framework

To address these issues, we introduce stableKT, a framework designed for robust "train-short, test-long" generalization. The architecture consists of several key modules that integrate linear biases and hyperbolic geometry.

The stableKT architecture featuring Hyperbolic and Dot-product Attention modules followed by a Length Generalization Module.

Length Generalization Module

Instead of relying on learned or sinusoidal position embeddings, stableKT utilizes a technique inspired by ALiBi (Attention with Linear Biases). It adds a non-learned penalty to attention scores proportional to the distance between the current question (query) and past interactions (keys). This shift from absolute position to relative distance allows the model to remain robust across any sequence length.

Hyperbolic Attention and Multi-head Aggregation

Educational concepts are inherently hierarchical. To capture this, we map student interactions into hyperbolic space (specifically the Poincaré half-space model), which is mathematically optimized for representing tree-like structures. The model employs a multi-head aggregation approach: some heads use standard dot-product attention to capture local patterns, while others use hyperbolic attention to identify broad, hierarchical relationships (e.g., connecting a specific problem to its broader "ancestor" concept).

Experimental Results

We evaluated stableKT on three major datasets: AL2005 (Algebra), BD2006 (Bridge to Algebra), and NIPS34 (NeurIPS 2020 Education Challenge).

Stability and Accuracy

While baseline models like simpleKT experienced AUC drops of up to 5.55% when moving from 200 to 1,000 interactions, stableKT maintained nearly constant performance. As shown in the comparison of position embedding methods, our Length Generalization Module provides a significantly flatter performance curve compared to Sinusoidal, Rotary, or T5 embeddings.

Comparison of AUC performance across different sequence lengths for various position embedding methods on three datasets.

In terms of raw accuracy, stableKT outperformed state-of-the-art baselines (including SAKT, SAINT, and AKT) across all datasets. On the AL2005 dataset at a sequence length of 1,000, it improved the AUC of the previous best model by nearly 7%.

Discussion and Conclusion

stableKT demonstrates that linear biases and hyperbolic geometry can effectively solve the length generalization crisis in Knowledge Tracing. By training on short, computationally efficient sequences, the model remains highly accurate for long student histories, facilitating more efficient large-scale deployments.

Future work may explore the model's performance in domains where knowledge is represented as directed acyclic graphs (DAGs) rather than strict trees, and investigate the integration of additional data types, such as student forum interactions or video viewing patterns.

Summary updated

Abstract

Knowledge tracing (KT) is the task of predicting students' future performance based on their historical learning interaction data. With the rapid advancement of attention mechanisms, many attention based KT models are developed. However, existing attention based KT models exhibit performance drops as the number of student interactions increases beyond the number of interactions on which the KT models are trained. We refer to this as the length generalization of KT model. In this paper, we propose stableKT to enhance length generalization that is able to learn from short sequences and maintain high prediction performance when generalizing on long sequences. Furthermore, we design a multi-head aggregation module to capture the complex relationships between questions and the corresponding knowledge components (KCs) by combining dot-product attention and hyperbolic attention. Experimental results on three public educational datasets show that our model exhibits robust capability of length generalization and outperforms all baseline models in terms of AUC. To encourage reproducible research, we make our data and code publicly available at https://pykt.org.
BibTeX
@inproceedings{li2024enhancing,
  author = {Xueyi Li and Youheng Bai and Teng Guo and Zitao Liu and Yaying Huang and Xiangyu Zhao and Feng Xia and Weiqi Luo and Jian Weng},
  title = {{Enhancing Length Generalization for Attention Based Knowledge Tracing Models with Linear Biases}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=A4mjPnEaMs},
}
Figure