Enhancing Length Generalization for Attention Based Knowledge Tracing Models with Linear Biases

Xueyi Li, Youheng Bai, Teng Guo, Zitao Liu

Abstract

Knowledge Tracing (KT) is a critical task in educational technology, yet Transformer-based models often suffer from a "length generalization" problem, where predictive accuracy declines as student interaction histories grow beyond training lengths. We present stableKT, a framework that ensures stable performance by integrating a Length Generalization Module using linear attention biases and a Hyperbolic Attention Module to capture hierarchical knowledge structures. Evaluations on AL2005, BD2006, and NIPS34 datasets demonstrate that stableKT maintains state-of-the-art AUC performance even as sequences scale to 1,000 steps, effectively solving the performance degradation seen in existing models.

Introduction

Predicting student performance based on past interactions is the cornerstone of personalized learning. While self-attention models like SAKT and SAINT have advanced the state-of-the-art, they typically struggle with long-range dependencies and length generalization. Most models are trained on short sequences (e.g., 200 interactions) and fail when applied to longer histories. Additionally, standard Euclidean attention is ill-suited for the tree-like hierarchical relationships inherent in educational curricula, such as the prerequisite relationship between basic arithmetic and multiplication.

The stableKT Framework

The stableKT architecture addresses these challenges through two primary innovations: hyperbolic embeddings for hierarchy and linear biases for length stability.

The stableKT architecture, featuring Hyperbolic and Dot-product Attention modules followed by a Length Generalization Module.

Hyperbolic Attention Module

To better model the hierarchical nature of knowledge, stableKT maps interactions into hyperbolic space. Specifically, it utilizes the Poincaré half-space, which provides a more natural geometry for representing tree structures than Euclidean space. This allows the model to focus on foundational "ancestor" concepts when necessary, while standard dot-product attention heads focus on local, recent interactions.

Length Generalization Module

Instead of traditional position embeddings, stableKT employs a Length Generalization (LG) module. It applies a linear penalty to attention scores based on the distance between queries and keys. This prevents the model from overfitting to specific sequence positions and allows it to handle histories of arbitrary length without retraining, avoiding the performance "cliff" seen in models like simpleKT.

Experimental Evaluation

We conducted extensive experiments on three benchmark datasets: AL2005, BD2006, and NIPS34, comparing stableKT against several baselines across sequence lengths ranging from 200 to 1,000.

Performance Stability

A key result of our study is the elimination of the performance degradation observed in baseline models. While models like simpleKT and SAINT see AUC drops of over 5% when moving from 200 to 1,000 interactions, stableKT maintains a nearly flat performance curve.

Comparison of AUC performance across different sequence lengths for various position encoding methods. stableKT (Length Generalization Module) shows superior stability.

As shown in the evaluation across datasets, stableKT consistently outperforms other position encoding methods like Sinusoidal, Rotary, and T5 embeddings, which all show significant degradation as sequence length increases. On the AL2005 dataset, stableKT achieved an AUC of 0.8348 at a sequence length of 1,000, significantly outperforming simpleKT (0.7655).

Conclusion

stableKT addresses the critical limitation of length generalization in Knowledge Tracing. By combining hyperbolic attention for hierarchical modeling with linear biases for sequence stability, the framework provides a robust and scalable solution for long-term student modeling. Future work will investigate the model's performance on more complex, non-hierarchical knowledge graphs and its computational efficiency in real-time applications.

Summary updated

BibTeX
@inproceedings{li2024enhancing,
  author = {Xueyi Li and Youheng Bai and Teng Guo and Zitao Liu},
  title = {{Enhancing Length Generalization for Attention Based Knowledge Tracing Models with Linear Biases}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=A4mjPnEaMs},
}
Figure