Aligning Learning Outcomes to Learning Resources: A Lexico-Semantic Spatial Approach

Swarnadeep Saha, Malolan Chetlur, Tejas Indulal Dhamecha, K. Gayathri Wijayarathna, Red Mendoza, Paul Gagnon, Nabil Zary, Shantanu GodboleIJCAI 2019 · DOI

TL;DR

This paper tackles the challenge of automatically connecting learning outcomes (LOs) to relevant sections within educational slide decks (Learning Resources or LRs). They introduce a novel "Lexico-Semantic Spatial" approach that combines text analysis with the sequential nature of slides, significantly improving navigation for students compared to previous methods. This work is crucial for making educational materials more accessible and efficient for problem-based learning.

What Problem Does This Paper Address?

In problem-based learning, students often need to consult various learning materials to solve a given problem. Manually sifting through lengthy documents or slide decks to find the specific information relevant to a particular learning objective can be time-consuming and inefficient. Learning Outcomes (LOs) are concise statements that define what students should know or be able to do after a learning activity. Learning Resources (LRs), such as slide decks, contain the information needed to achieve these LOs. The core problem is to automatically identify which pages within an LR are most pertinent to a given LO. This is particularly challenging because slide decks are often brief, visually oriented, and their content might not always be strictly ordered by topic. Existing document retrieval methods often treat pages as independent documents, failing to leverage the sequential and contextual information present in a slide deck.

Methods

The authors propose a novel three-pronged approach called "Lexico-Semantic Spatial" to tackle this problem. It's a pipelined approach that first identifies page relevance and then uses this to align LOs to sets of pages.

  1. Lexical Model: This model identifies relevant pages by looking for direct word overlaps between the LO and the title/body of a slide. It uses simple overlap counts.
  2. Semantic Model: This model goes beyond exact word matches. It uses a sophisticated text encoder, specifically a Bidirectional Long Short-Term Memory (BiLSTM) network with a self-attention mechanism. This encoder processes the LO, slide title, and slide body to create dense vector representations. These representations are then used to measure semantic similarity, capturing meaning even when different words are used.
  3. Spatial Model: This is the most novel aspect. It recognizes that the order of slides matters. This model takes the relevance scores from the lexical and semantic models and uses them to infer the final relevance. It considers features like the relevance of neighboring pages, the range of relevance scores across the entire slide deck, and the distribution of relevance scores (histogram). This helps to identify contiguous chunks of relevant pages and smooth out relevance predictions.

The overall approach is designed to be extensible and to work even with limited training data.

Datasets

The authors created a dataset from a medical curriculum, consisting of 100 Learning Resources (LRs) in the form of slide decks. This dataset includes 694 Learning Outcomes (LOs) and expert annotations identifying which pages are relevant to each LO. This dataset is used for both training and evaluating their models.

Key Insights and Ideas

  • The importance of spatial context: Simply analyzing each slide in isolation (lexically or semantically) is insufficient. The sequential arrangement of slides and their proximity to other relevant content significantly impacts a slide's overall relevance to an LO.
  • Leveraging limited data: The proposed pipelined approach, especially by breaking down the problem into a page-relevancy classification task, effectively addresses the challenge of scarce annotated data.
  • "Click" metric as a user-centric evaluation: Beyond traditional precision/recall, the "click" metric provides a more intuitive measure of how easily a student can navigate to relevant information, simulating real-world user experience.
  • Spatial model's dual role: The spatial model not only helps fill in gaps between relevant pages but also helps remove isolated, irrelevant pages, leading to more coherent and useful sets of recommended pages.

Key Results

  • Page Relevancy Task: The full Lexico-Semantic Spatial approach achieved the best macro-F1 score (0.6609), outperforming individual lexical or semantic models, as well as versions augmented with only spatial features.
  • LO Alignment Task: The Lexico-Semantic Spatial approach achieved a significantly better F1 score (0.4171) compared to lexical (0.3375) and semantic (0.4026) models alone. With post-processing, it reached 0.4464.
  • Navigation Efficiency (Click Metric): The most significant result is the improvement in student navigation. Their best model allows a student to reach a relevant page in an average of 4 clicks from the first presented page, compared to a baseline of 18 clicks and human performance of 2 clicks for a typical 38-page slide deck. This demonstrates a substantial improvement in learning efficiency.
  • Chunk and Span Reduction: The spatial model effectively reduced the number of "chunks" (contiguous sets of relevant pages) and the overall "span" (range of pages) for recommended content, making it more focused.

Limitations and Future Work

  • Image Content Ignored: A major limitation acknowledged is that the current approach does not exploit information within images on the slides. This leads to a significantly lower recall, as many relevant pages might rely heavily on visual content.
  • Text Encoder for Slides: The authors note that the text encoder in the semantic model could be improved, as slide text is often non-sentential and lacks a definite order, posing challenges for standard sequence models.
  • Future Directions:
    • Extend the approach to incorporate image analysis for better recall.
  • Improve the text encoder to better handle the unique characteristics of slide text.
  • Validate the generic nature of the approach by applying it to other document types and question-answering tasks.

Summary updated

BibTeX
@inproceedings{saha2019aligning,
  author = {Swarnadeep Saha and Malolan Chetlur and Tejas Indulal Dhamecha and K. Gayathri Wijayarathna and Red Mendoza and Paul Gagnon and Nabil Zary and Shantanu Godbole},
  title = {{Aligning Learning Outcomes to Learning Resources: A Lexico-Semantic Spatial Approach}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/718},
}
Figure