Abstract
This paper introduces backward model tracing, a novel student modeling methodology designed to reconstruct a student's reasoning process by working in reverse from their final answer. Implemented within the English Tutor (ET) system—designed to help Italian students learn the English verb system—the approach combines pre-defined bug libraries with Explanation-Based Learning (EBL) to dynamically generate error rules. By reverse-engineering the student's logic through a series of expert processing phases, the system provides a deep, non-intrusive diagnosis of misconceptions in complex domains like linguistic conjugation.
Introduction
In the field of Intelligent Tutoring Systems (ITS), accurately modeling a student's knowledge and misconceptions is critical for providing effective feedback. Traditional approaches, such as overlay models or static bug collections, often fail to capture the underlying logic of a student's error or require exhaustive manual labor to build. Furthermore, real-time model tracing can be intrusive, interrupting the student's problem-solving flow.
To address these challenges, the authors propose backward model tracing. This method allows the tutor to analyze a final incorrect answer and derive the sequence of (potentially flawed) reasoning steps that led to it. By focusing on the "why" behind an error, the system can distinguish between simple slips and deep conceptual misunderstandings.
Methodology: The ET System Architecture
The core of this approach is implemented in the English Tutor (ET). The expert knowledge in ET is modularized into five distinct phases, as shown in Figure 1.
When a student provides an answer, the system performs a Reverse Reconstruction. It works backward through these five phases—Conjugation (P5), Tense Selection (P4), Reference Time (P3), Temporal Relations (P2), and Parsing (P1)—to find a path that explains the student's output.
Hybrid Bug Modeling and EBL
The system employs a hybrid strategy for error diagnosis:
- Bug Library: A collection of common, pre-defined errors.
- Meta-Bug Library: High-level rules that describe how to perturb expert rules (e.g., substituting the auxiliary verb "be" for "have").
- Explanation-Based Learning (EBL): When a Meta-Bug identifies a new type of error, EBL is used to generalize this specific instance into a reusable "malrule." This allows the system to learn a student's unique patterns from very few examples.
To manage multiple potential explanations for a single error, the system utilizes a Truth Maintenance System (TMS). The TMS tracks various diagnostic hypotheses and refines them as the student provides more data over time, discarding unlikely theories that fail to predict future behavior.
Reconstructing Student Reasoning
The primary output of the backward tracing process is a derivation tree. This tree serves as a map of the student's logic, showing exactly where they diverged from the expert path. Figure 2 illustrates a derivation tree for the incorrect conjugation "is studyed" (instead of "has studied").
In this example, the system identifies both a conceptual error (using the auxiliary "be" instead of "have" via meta-bug MB1) and a morphological error (incorrectly applying a regular verb rule to create "studyed" via malrule MR1). This level of diagnostic depth is achieved without interrupting the student during the task.
Key Results and Insights
The methodology was successfully implemented in QUINTUS PROLOG on a SUN 3 workstation. The evaluation demonstrated that the integration of bug collection and bug construction effectively overcomes the limitations of using either technique in isolation. Key insights include:
- Diagnostic Depth: The system can pinpoint the exact phase where reasoning failed, distinguishing between surface-level spelling and deep conceptual errors.
- Efficiency: EBL allows the system to model complex misconceptions without needing massive datasets, often learning from a single mistake.
- Non-Intrusiveness: Post-hoc analysis maintains the student's focus while providing high-quality diagnostic data comparable to real-time monitors.
Limitations and Future Work
While powerful, backward model tracing relies on the assumption that students follow the same general cognitive phases as the expert. If a student employs a completely different, non-linear strategy, the system may struggle to map the reasoning. Additionally, maintaining a TMS with many competing hypotheses can be computationally intensive as sessions progress.
Future research aims to improve the tutor module's "active sensing" capabilities, allowing it to select specific exercises that help resolve ambiguities between competing diagnostic hypotheses in the student model.
Summary updated