Using Learning Analytics for Adaptive Exercise Generation

Tanja Heck, Detmar MeurersACL 2023

Sure, here's a blog-style summary of the paper "Using Learning Analytics for Adaptive Exercise Generation":

TL;DR

This paper presents a novel approach to automatically generate adaptive language learning exercises, specifically focusing on Single Choice (SC) questions. By leveraging learning analytics and an existing error diagnosis system, the authors demonstrate how to dynamically create and adapt distractors that target individual learner misconceptions, making exercises more effective and personalized.

What Problem Does This Paper Address?

Creating effective language learning exercises, especially those that challenge learners within their "zone of proximal development," is a complex task. Single Choice (SC) exercises, while common, pose a particular challenge: generating plausible distractors (incorrect answer options) that are pedagogically sound. These distractors should reflect common learner errors and misconceptions without introducing entirely novel, unhelpful ones. Manually creating these distractors is time-consuming and often doesn't account for individual learner differences. Existing automated methods often rely on static analyses of learner corpora, which can't adapt to dynamic learning needs. This paper addresses the gap of creating dynamic, learner-adaptive distractors for SC exercises by integrating learning analytics with error diagnosis.

Methods

The core of the paper's methodology lies in repurposing a system's error diagnosis module for adaptive exercise generation. Instead of relying on static learner data, the authors use dynamic learning analytics derived from "half-open" exercises (where learners produce a specific form) to inform the generation of distractors for "closed" SC exercises.

Key steps include:

  • Error Diagnosis and Annotation: Utilizing an existing error diagnosis module to identify and label learner errors in half-open exercises. This is complemented by manual annotation to ensure comprehensive coverage and refine labels.
  • Learning Goal Identification: Grouping related error labels into pedagogically meaningful learning goals.
  • Distractor Generation: Using the identified frequent learner errors from half-open exercises as candidates for distractors in SC exercises. The system aims to select distractors that reflect common misconceptions.
  • Sentence Chunking for Jumbled Sentences: Applying the same error analysis to identify constituents that should be separated into chunks for Jumbled Sentence exercises, aiding in word order practice.

Datasets

The study utilizes data collected from the Interact4School (I4S) project. This dataset comprises learner interactions with an Intelligent Tutoring System (ITS) over a school year, specifically focusing on exercises related to the simple past tense in English. The data includes learner answers to four types of exercises: Jumbled Sentences (JS), Single Choice (SC), Fill-in-the-Blanks (FiB), and Short Answer (SA). A total of 4,836 incorrect learner answers from 199 learners were analyzed.

Key Insights and Ideas

  • Dynamic Distractors are Key: The most significant insight is that dynamic learning analytics, derived from how learners actually err in practice, can be a powerful source for generating effective distractors. This moves beyond static corpus analysis.
  • Error Diagnosis as a Foundation: An existing error diagnosis module, typically used for feedback, can be effectively repurposed for distractor generation, highlighting the potential for synergy between different ITS components.
  • Human Intuition vs. Data-Driven: While there's overlap, human exercise creators' focus on misconceptions can differ from what learning analytics reveal as most frequent. This suggests a hybrid approach where humans define learning goals, and analytics drive generation.
  • Exercise Type Matters: Not all exercise types are equally effective for practicing all types of errors. For instance, omission errors are difficult to represent in SC exercises.

Key Results

  • The system's error diagnosis module successfully identified a significant portion (34 out of 63) of relevant error labels for simple past questions.
  • The analysis showed that frequent errors in half-open exercises often correspond to plausible distractors in SC exercises, supporting the core hypothesis.
  • For auxiliary verbs, the most frequent error (omission) is not well-covered by SC distractors, but other common errors (e.g., using "be" or "have" instead of "do") are.
  • For main verbs, distractors cover common errors like using the wrong past form, simple present, or incorrectly formed variants.
  • The study found that while distractors generally align with common learner errors, some manually created distractors were less plausible than expected, and some common errors were not covered by system-generated distractors.

Limitations and Future Work

  • Incomplete Coverage: The automatic annotations do not yet cover all identified misconceptions, particularly for main verbs and questions with "be."
  • Exercise Type Limitations: Certain errors (like omission) are inherently difficult to represent in specific exercise types (e.g., SC).
  • Pedagogical Validity of Distractors: While the study aimed for pedagogical validity, the coverage of distractors for all misconceptions is not complete.
  • Future Work: The authors suggest further research to determine if errors learners make in half-open exercises are also good distractors for SC exercises, and to investigate how misconceptions evolve over time within and across learners. They also highlight the need to extend the error diagnosis module's coverage.

Summary updated

BibTeX
@inproceedings{heck2023using,
  author = {Tanja Heck and Detmar Meurers},
  title = {{Using Learning Analytics for Adaptive Exercise Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure