Abstract
Automated, data-driven feedback systems are becoming staples in introductory computer science education, yet their evaluation remains largely confined to surface-level metrics like accuracy and task completion time. This research proposes a comprehensive five-criteria framework to assess these systems: computational accuracy, guidance, effectiveness, the impact of fallibility, and generalized student response. By analyzing novice programmers using the Data-Driven Adaptive Immediate Feedback (DDAIF) system within the iSnap environment, this study explores the human-AI interaction loop, specifically focusing on how students react when automated feedback is incorrect. Results indicate that while systems can improve persistence, their fallibility often leads to over-reliance, where students stop self-evaluating in favor of "tutor-seeking" behavior.
Introduction
As introductory computer science courses scale, data-driven feedback systems—which offer hints by comparing current student code to historical data—have emerged as a scalable solution for providing immediate support. However, these systems are inherently fallible; they often struggle with novel or creative solutions that deviate from the training data. Traditional evaluations often overlook the consequences of these errors. This paper addresses this gap by investigating the "Check Engine" light of programming: what happens when the automated tutor is wrong? We seek to understand how false positives and false negatives influence student behavior and learning outcomes.
Methodology
The study evaluated the DDAIF system integrated into iSnap, a block-based programming environment. The evaluation involved 27 non-CS majors completing a "Squiral" (square-spiral) assignment. The researchers employed a five-criteria framework to measure computational accuracy, guidance, effectiveness, the impact of fallibility, and generalized student response. To analyze student progress, the team utilized State Transition Diagrams (STDs), which visualize the paths students take through various code states toward a solution.
The Feedback Interface and Task
The "Squiral" task requires students to create a custom block that draws a spiral of squares. The DDAIF system provides real-time feedback via a checklist of objectives, indicating whether specific components of the code (like the length of the spiral increasing with each side) have been correctly implemented.
Key Results and Analysis
The system achieved a computational accuracy of 76%. While the "True Positive" rate (identifying finished steps) was 83%, the "True Negative" rate (identifying incomplete steps) dropped to 61%. This discrepancy highlights a significant risk: students may be told they have completed a step when they have not.
The Reliance Trap and Negative Impacts
A primary finding was the "Reliance Trap." Novice programmers often deferred their own judgment to the system. If a progress bar turned green, students assumed correctness regardless of their program's actual output. The researchers categorized the fallout of faulty feedback into three types:
- Impact on Programming Behavior (IPB): Retaining incorrect code due to false validation.
- Impact on Time Spent (ITS): Wasting time "fixing" correct code due to false negatives.
- Early Submission (ES): Submitting incomplete work after receiving premature validation.
Notably, 40% of faulty feedback instances resulted in a documented negative impact. False positives were particularly detrimental, with a 78.5% impact ratio.
Discussion and Future Directions
While the feedback system improved overall persistence—with 93% of students completing the task—it also introduced "lingering impacts" where early errors derailed students for significant periods.
Future development should move away from binary "correct/incorrect" indicators toward systems that communicate their own fallibility. For instance, a system might provide a confidence score (e.g., "70% certain this step is complete") to encourage continued student self-evaluation. Furthermore, transitioning from pattern-matching to semantics-based feedback could improve accuracy by understanding the underlying logic of student code rather than just its surface structure.
Summary updated