Generating Feedback for English Foreign Language Exercises

Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy, Verena Möller, Florian Nuxoll, Detmar MeurersNAACL 2018

TL;DR

This paper introduces a novel approach to generating immediate, metalinguistic feedback for foreign language grammar exercises. By pre-generating a comprehensive set of expected correct and incorrect answers based on task information, their system can efficiently provide targeted feedback to learners, addressing a key challenge in automated language learning systems.

What Problem Does This Paper Address?

Providing timely and individualized feedback is crucial for language learning, but it's a significant challenge for human teachers in large classrooms. While Intelligent Language Tutoring Systems (ILTS) aim to automate this, many struggle with providing truly helpful, metalinguistic feedback. Traditional ILTS often rely on analyzing learner errors directly (e.g., "mal-rules") or use flexible mechanisms that can be computationally expensive and may not consider the learner's intended meaning or the specific task context. This can lead to misleading feedback, as illustrated by an example where a system incorrectly suggests changing a word instead of explaining a grammatical form error. The authors aim to bridge this gap by integrating task-specific information into the feedback generation process, making it more precise and efficient.

Methods

The core of their approach is a two-part system:

  • Offline Hypothesis Generation: This is the novel part. Instead of analyzing learner errors directly, they start with the correct target answers for an exercise. Using a suite of NLP tools (segmentation, POS tagging, lemmatization, etc.), they create a rich linguistic representation of the target answer. Then, they apply a series of custom rules (organized in "rule layers" inspired by graph search algorithms) to systematically generate a wide range of both well-formed variations and plausible ill-formed variations that a learner might produce. Each generated hypothesis is paired with a diagnosis of the change made. This process is performed before the learner interacts with the system.
  • Online Matching: When a student submits an answer, the system uses a flexible matching approach to compare it against the pre-generated hypotheses. This involves a multi-fallback strategy, starting with direct matches and progressing to more flexible methods like partial matching and information retrieval (using Lucene) to account for paraphrases and minor variations. If a match is found, the associated diagnosis and feedback template are used to generate the metalinguistic feedback.

Datasets

The study uses data collected from a web-based online workbook called FeedBook, implemented in German secondary schools for 7th-grade English learners. The analysis is based on a snapshot of data collected from October 2016 to March 2018, comprising 6341 submissions of complete exercises by 538 students. For the evaluation, they used 33,589 individual student answers from 111 feedback-enabled tasks (64 short answer, 47 fill-in-the-blanks).

Key Insights and Ideas

  • Shifting the Paradigm: The most significant insight is moving from analyzing learner errors to generating expected variations from correct answers. This "generate-and-retrieve" approach leverages linguistic knowledge of the target language more directly.
  • Task-Informed Feedback: Explicitly incorporating task information (what the exercise is about) is crucial for generating relevant and accurate feedback, preventing misinterpretations.
  • Pre-computation Efficiency: Performing hypothesis generation offline avoids performance bottlenecks during real-time interaction, allowing for faster and more precise feedback delivery.
  • The "Long Tail" Problem: While the system excels at common error patterns (especially tense variations), a significant portion of student answers represent infrequent deviations. This highlights the need for robust fallback strategies and potentially meaning-based analysis for these cases.
  • Teacher Collaboration: The development process actively involved teachers, ensuring that the feedback generated is pedagogically sound and aligns with real-world classroom practices.

Key Results

  • The system successfully covers frequent answer patterns, with specific, metalinguistic feedback provided for 25.28% of the differing answers.
  • Tense-related problems were frequently diagnosed, which teachers identified as a challenging topic.
  • For specific feedback, the ratio of instances to types was approximately 1:5 (3,483 instances for 696 types), indicating that modeling specific error patterns is worthwhile.
  • Default feedback (showing the difference between the student's answer and the target) was provided for the majority of differing answers (74.72%).
  • Analysis of edit distances suggests that for answers with greater variation (paraphrases, meaning errors), meaning assessment approaches are needed.

Limitations and Future Work

  • Meaning vs. Form: The current system primarily focuses on form-based feedback. For answers that diverge due to meaning errors or paraphrases, the system currently provides default feedback.
  • Infrequent Deviations: The "long tail" of rare error types is challenging to model with specific feedback.
  • Future Work:
    • Integrate meaning assessment using approaches like CoMiC.
  • Strengthen task modeling to better determine optimal feedback strategies.
  • Develop learner modeling to track individual error histories and suggest appropriate next tasks.
  • Address the absence of specific grammatical forms (e.g., missing gerunds).
  • Conduct a randomized controlled field study to assess actual learning gains.

Summary updated

BibTeX
@inproceedings{rudzewitz2018generating,
  author = {Björn Rudzewitz and Ramon Ziai and Kordula De Kuthy and Verena Möller and Florian Nuxoll and Detmar Meurers},
  title = {{Generating Feedback for English Foreign Language Exercises}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure