Deep Factorization Machines for Knowledge Tracing

Jill-Jênn VieNAACL 2018

TL;DR

This paper introduces a novel application of Deep Factorization Machines (DeepFM) for knowledge tracing, a task focused on predicting student performance on educational content. By effectively combining user, item, and skill information with rich side features, their model achieved competitive results on the 2018 Duolingo Shared Task, demonstrating the power of deep learning architectures for personalized education.

What Problem Does This Paper Address?

Personalizing education is a major goal, aiming to provide students with tailored exercises and lessons based on their current knowledge. This requires accurate student modeling, which involves inferring a student's understanding from their interactions with learning materials. The 2018 Duolingo Shared Task presented a specific challenge: predicting whether a student would correctly answer words in a new sentence, given a wealth of side information about the sentences themselves (lexical, morphological, syntactic) and student interactions. Traditional knowledge tracing models often struggle to incorporate such rich, diverse features, leading to the exploration of more advanced machine learning techniques.

Methods

The core of this paper's approach is the Deep Factorization Machine (DeepFM). DeepFM is a hybrid model that combines the strengths of Factorization Machines (FM) and deep neural networks (DNNs).

  • Factorization Machine (FM) Component: This part models pairwise interactions between features. It's particularly good at capturing relationships between different entities (like users and items) and can even represent simpler models like logistic regression and the Rasch model as special cases.
  • Deep Component: This is a standard feedforward neural network that learns higher-order feature interactions. It takes embeddings of the activated entities as input.

The authors adapted DeepFM to the knowledge tracing task by treating each instance (a student's attempt at a word) as a sparse vector. They then trained the model to predict the probability of a correct answer. The model learns embeddings for various entities (user, item, skill, etc.) and leverages both the FM and DNN components to make predictions.

Datasets

The primary dataset used in this paper is from the 2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM). This dataset consists of thousands of student attempts on sentences, annotated with various features. The authors also mention using a validation set for model tuning and then evaluating on a test set.

Key Insights and Ideas

  • DeepFM as a Unified Framework: The paper highlights how DeepFM can elegantly unify and extend various existing student modeling approaches, from simple logistic regression to more complex models, by treating them as special cases within its architecture.
  • Value of Side Information: The results suggest that incorporating rich side information (lexical, morphological, syntactic features of sentences) alongside fundamental user and item data can significantly improve prediction accuracy.
  • Feature Engineering vs. End-to-End Learning: While DeepFM* (using all features) performed slightly worse than DeepFM (using only fundamental features), it points to the potential of richer feature sets, but also the importance of how these features are integrated and potentially the need for more sophisticated feature interaction modeling (e.g., field-aware approaches).
  • Embeddings for Sparse Data: The approach demonstrates how to learn dense embeddings from sparse features commonly found in educational platforms, which is crucial for effective machine learning.

Key Results

  • The authors' DeepFM solution achieved an AUC of 0.815 on the en_es dataset for the 2018 Duolingo Shared Task.
  • This performance improved upon the logistic regression baseline (AUC 0.772) by approximately 3 percentage points in AUC.
  • The best performing model was DeepFM using only discrete fundamental features, achieving an AUC of 0.815.
  • DeepFM* (using all features, including noisy and continuous ones) performed slightly worse, suggesting that feature selection and integration are critical.

Limitations and Future Work

  • Feature Integration Complexity: The authors note that their DeepFM* model, which incorporated all features, performed slightly worse than the model using only fundamental features. They suggest that this might be mitigated by using field-aware factorization machines, which can learn specific parameters for different feature categories to better manage their importance.
  • Dependency Graph Embedding: They propose embedding the dependency graph of sentences into the vanilla FM model as a future direction.
  • Ensemble Methods: The paper suggests exploring ensemble methods like XGBoost, which are common in competitive machine learning challenges, to further boost performance.

Summary updated

BibTeX
@inproceedings{vie2018deep,
  author = {Jill-Jênn Vie},
  title = {{Deep Factorization Machines for Knowledge Tracing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure