Unsupervised Cross-Domain Prerequisite Chain Learning using Variational Graph Autoencoders

Irene Li, Vanessa Yan, Tianxiao Li, Rihao Qu, Dragomir R. Radev

TL;DR

This paper introduces a novel method, CD-VGAE, for learning prerequisite relationships between concepts across different domains without requiring labeled data in the target domain. By leveraging a variational graph autoencoder and incorporating "domain neighbors" (concepts shared or similar across domains), the model effectively transfers knowledge from a data-rich source domain to a data-scarce target domain, outperforming existing methods.

What Problem Does This Paper Address?

In today's vast digital learning landscape, efficiently acquiring new knowledge is crucial. Understanding the prerequisite relationships between concepts (e.g., knowing that "Convolutional Neural Networks" is a prerequisite for "Image Classification") helps create structured learning paths. However, learning these relationships is challenging, especially when moving to a new domain where such knowledge is scarce. Traditional methods often rely on labeled data within the target domain or extract information directly from content, which can be labor-intensive. This paper tackles the problem of unsupervised cross-domain prerequisite chain learning, aiming to transfer learned prerequisite knowledge from a well-understood domain (like NLP) to a less understood one (like Computer Vision) without explicit supervision in the target domain.

Methods

The core of the proposed solution is the Cross-Domain Variational Graph Autoencoder (CD-VGAE). This model builds a heterogeneous graph that connects concept and resource nodes from both source and target domains. The graph's adjacency matrix is constructed using four types of edges:

  • As: Edges between source concept nodes.
  • Arc: Edges between resource and concept nodes.
  • Ar: Edges between resource nodes.
  • Act: Edges between target concept nodes.

The model uses a Graph Convolutional Network (GCN) encoder to learn node representations. A key innovation is the introduction of "domain neighbors". These are concepts from the source domain that are semantically similar to concepts in the target domain, identified using cosine similarity on their embeddings. This allows the model to leverage shared knowledge. The encoder incorporates information from both direct neighbors and these domain neighbors. For the decoder, the authors adapt the DistMult method to predict the likelihood of a prerequisite link between concept pairs. The entire process is unsupervised, meaning it learns these relationships without explicit labels in the target domain.

Datasets

The authors utilize and expand upon the LectureBank 2.0 dataset.

  • Source Domain (NLP): The original LectureBank 2.0 dataset, containing 1,717 lecture slides and 322 concepts with annotated prerequisite relations.
  • Target Domains (CV and BIO): The paper introduces an expansion called LectureBankCD, which includes two new domains: Computer Vision (CV) and Bioinformatics (BIO). For these domains, they collected high-quality lecture slides and annotated concepts and prerequisite relations. The CV domain has 201 concepts with 871 positive relations, and the BIO domain has 100 concepts with 234 positive relations. The data and code are made publicly available.

Key Insights and Ideas

  • Leveraging "Domain Neighbors" is Crucial: The most significant insight is that explicitly incorporating semantically similar concepts from the source domain (domain neighbors) into the graph learning process dramatically improves the transfer of prerequisite knowledge to the target domain.
  • Unsupervised Cross-Domain Transfer is Feasible: The paper demonstrates that it's possible to learn meaningful prerequisite relationships in a new domain without any labeled data from that domain, relying solely on knowledge from a related, richer domain.
  • Graph-based Models Excel at Capturing Relationships: Unlike simple classifier-based approaches that treat concept pairs independently, graph-based models like VGAE and CD-VGAE can better capture the interconnectedness of concepts within a larger knowledge structure.
  • P2V Embeddings Outperform BERT for This Task: For node embeddings, the authors found that embeddings trained on the resource data (P2V) generally yielded better results than pre-trained BERT embeddings, possibly due to BERT's high dimensionality leading to overfitting.

Key Results

  • Substantial Performance Gains: The proposed CD-VGAE model significantly outperforms baseline methods (including simple classifiers and standard VGAE) on both CV and BIO target domains in terms of accuracy, F1 score, and especially recall. For instance, in the CV domain, CD-VGAE achieved an F1 score of 0.6754, compared to 0.5757 for VGAE+P2V.
  • High Recall is a Key Advantage: The model demonstrates a notably higher recall compared to baselines, which is crucial for educational applications where missing relevant concepts can hinder learning.
  • Qualitative Improvements: Case studies show that CD-VGAE recovers more accurate and complete prerequisite chains. For example, in the BIO domain, it correctly identifies more prerequisite concepts for "BLAST" than VGAE. In the CV domain, it recovers nearly all direct neighbors for "Image Processing" and "Object Localization," significantly outperforming baselines that miss many concepts.

Limitations and Future Work

  • Limitations of Embeddings: While P2V embeddings performed better, the paper notes that BERT embeddings might be prone to overfitting due to their high dimensionality.
  • GraphSAGE Performance: The authors experimented with GraphSAGE for out-of-domain transfer but obtained negative results, suggesting that the standard inductive GraphSAGE might not be directly suitable for this specific cross-domain task. They leave further investigation of inductive methods as future work.
  • Future Directions: The paper suggests exploring other graph-based methods and further investigating inductive graph learning approaches for cross-domain transfer. The creation of a new, publicly available dataset (LectureBankCD) also opens avenues for future research in this area.

Summary updated

Abstract

Learning prerequisite chains is an essential task for efficiently acquiring knowledge in both known and unknown domains. For example, one may be an expert in the natural language processing (NLP) domain but want to determine the best order to learn new concepts in an unfamiliar Computer Vision domain (CV). Both domains share some common concepts, such as machine learning basics and deep learning models. In this paper, we propose unsupervised cross-domain concept prerequisite chain learning using an optimized variational graph autoencoder. Our model learns to transfer concept prerequisite relations from an information-rich domain (source domain) to an information-poor domain (target domain), substantially surpassing other baseline models. Also, we expand an existing dataset by introducing two new domains: CV and Bioinformatics (BIO). The annotated data and resources, as well as the code, will be made publicly available.
BibTeX
@inproceedings{li2021unsupervised,
  author = {Irene Li and Vanessa Yan and Tianxiao Li and Rihao Qu and Dragomir R. Radev},
  title = {{Unsupervised Cross-Domain Prerequisite Chain Learning using Variational Graph Autoencoders}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=0INsE04pOKW},
}
Figure