TL;DR
This paper tackles the challenge of automatically identifying relationships between learning outcomes (LOs) in a complex medical curriculum. By combining curriculum structure with semantic analysis of LO text and associated learning resources, they achieve human-level performance in classifying LO relatedness, paving the way for more intelligent educational tools.
What Problem Does This Paper Address?
Medical curricula, especially in fields like medicine, are vast and intricate. They are often organized hierarchically, with thousands of learning outcomes (LOs) spanning multiple years. For students to truly master the material, they need to understand how these LOs connect, not just within a single topic but across different parts of the curriculum. Manually identifying these connections is a monumental task, prone to errors and impossible to scale. This paper addresses the need for an automated system that can accurately discover these often-hidden relationships between LOs, enabling better curriculum design and more effective learning support.
Methods
The authors propose a cascaded classification approach to identify relationships between pairs of LOs. Their system leverages two main types of features:
- Curriculum-based Features: These capture the spatial and temporal proximity of LOs within the curriculum's hierarchical structure. Spatial proximity considers how close LOs are in terms of themes, fundamentals, and topics, while temporal proximity looks at when they are taught within the academic year.
- Semantic Features: These focus on the meaning of the LO text. They employ:
- Word Overlap: A soft matching of words between LOs using word embeddings, considering pairs of words whose cosine similarity exceeds a threshold.
- Histogram of Partial Similarities (HoPS): This method captures a more granular similarity profile by comparing each word in one LO to all words in another, creating histograms of similarity scores.
- Learning Resource (LR) Integration: Recognizing that LOs are often concise, the system augments LO text with content from their associated learning resources (e.g., slide decks) to enrich semantic analysis.
- Expert Medical Codes (EMC): They also incorporate domain-specific knowledge provided by medical experts, such as physiologic and pathologic states, to further refine the feature set.
The classification is performed in two stages: first, a binary classifier determines if two LOs are "Related" or "Unrelated." If deemed related, a second classifier distinguishes between "Strong" and "Weak" relationships. This cascaded approach is motivated by the observation that distinguishing between related and unrelated LOs is easier than differentiating between strong and weak relationships.
Datasets
The primary dataset used in this work is the curriculum from the Lee Kong Chian School of Medicine at Nanyang Technological University, Singapore. This curriculum comprises 4,251 LOs organized into a hierarchy. The authors also utilized expert-annotated data, where Subject Matter Experts (SMEs) labeled pairs of LOs as "Strong," "Weak," or "None." A subset of this annotated data was used to evaluate inter-annotator agreement.
Key Insights and Ideas
- Curriculum Structure and Semantics are Complementary: The paper strongly demonstrates that neither curriculum hierarchy nor semantic similarity alone is sufficient. Combining both provides a much richer understanding of LO relatedness, significantly outperforming systems relying on only one.
- Learning Resources are Crucial for Concise LOs: LOs are often very brief. Leveraging the text from associated Learning Resources (LRs) is essential to provide the necessary context for accurate semantic analysis and relationship extraction.
- Cascaded Classification is More Effective: The decision boundary between "Related" and "Unrelated" is more distinct than that between "Strong" and "Weak." A two-stage classification process effectively handles this, leading to better overall performance.
- Expert Medical Codes Don't Always Add Value: Surprisingly, adding expert-defined medical codes (physiologic/pathologic states) did not improve performance beyond using curriculum and semantic features. This suggests that the combined curriculum and semantic features might implicitly capture this information.
- Building Case Maps: The LO relationship extraction system can be directly applied to construct "case maps" for virtual patient scenarios, providing students with a comprehensive view of the LOs relevant to a specific clinical case.
Key Results
- The proposed cascaded classifier using a combination of curriculum and semantic features (CR+SM) achieved the best performance, with a macro-F1 score of 72.6%, approaching human-level performance (inter-annotator agreement of 69.9%).
- Integrating Learning Resource (LR) text improved LO-relationship extraction, with the cascaded CR+SM model achieving 67.2% macro-F1 on the LR-linked test set, compared to 65.9% without LRs.
- The system was successfully applied to generate a case map for a virtual dengue fever patient, which was evaluated by an SME and found to be largely accurate, demonstrating the practical utility of the approach.
Limitations and Future Work
- Inter-Annotator Disagreement: The authors note substantial disagreement between expert annotators, particularly for distinguishing between "Strong" and "Weak" relationships, highlighting the inherent subjectivity in this task.
- EMC Feature Utility: The inclusion of Expert Medical Codes (EMC) did not improve performance, suggesting further investigation into how this domain knowledge can be better integrated or if the existing features implicitly capture it.
- Future Work: The authors propose extending the concept map generation to cover all five years of the curriculum, characterizing relationships between edges (e.g., prerequisite), and investigating how the learned intermediate representations can be used to enrich curriculum metadata.
Summary updated