A Multicriteria Evaluation for Data-Driven Programming Feedback Systems: Accuracy, Effectiveness, Fallibility, and Students' Response

Preya Shabrina, Samiha Marwan, Andrew Bennison, Min Chi, Thomas W. Price, Tiffany BarnesarXiv 2022 · DOI · arXiv

Abstract

Automated, data-driven feedback systems are becoming staples in introductory computer science education, yet their evaluation remains largely confined to surface-level metrics like accuracy and task completion time. This research proposes a comprehensive five-criteria framework to assess these systems: computational accuracy, guidance, effectiveness, the impact of fallibility, and generalized student response. By analyzing novice programmers using the Data-Driven Adaptive Immediate Feedback (DDAIF) system within the iSnap environment, this study explores the human-AI interaction loop, specifically focusing on how students react when automated feedback is incorrect. Results indicate that while systems can improve persistence, their fallibility often leads to over-reliance, where students stop self-evaluating in favor of "tutor-seeking" behavior.

Introduction

As introductory computer science courses scale, data-driven feedback systems—which offer hints by comparing current student code to historical data—have emerged as a scalable solution for providing immediate support. However, these systems are inherently fallible; they often struggle with novel or creative solutions that deviate from the training data. Traditional evaluations often overlook the consequences of these errors. This paper addresses this gap by investigating the "Check Engine" light of programming: what happens when the automated tutor is wrong? We seek to understand how false positives and false negatives influence student behavior and learning outcomes.

Methodology

The study evaluated the DDAIF system integrated into iSnap, a block-based programming environment. The evaluation involved 27 non-CS majors completing a "Squiral" (square-spiral) assignment. The researchers employed a five-criteria framework to measure computational accuracy, guidance, effectiveness, the impact of fallibility, and generalized student response. To analyze student progress, the team utilized State Transition Diagrams (STDs), which visualize the paths students take through various code states toward a solution.

Figure 6: A State Transition Diagram (STD) mapping student progress through various code states toward completion.

The Feedback Interface and Task

The "Squiral" task requires students to create a custom block that draws a spiral of squares. The DDAIF system provides real-time feedback via a checklist of objectives, indicating whether specific components of the code (like the length of the spiral increasing with each side) have been correctly implemented.

Figure 2: The iSnap Block Editor showing the DDAIF feedback interface with a checklist of "Squiral Objectives."

Key Results and Analysis

The system achieved a computational accuracy of 76%. While the "True Positive" rate (identifying finished steps) was 83%, the "True Negative" rate (identifying incomplete steps) dropped to 61%. This discrepancy highlights a significant risk: students may be told they have completed a step when they have not.

The Reliance Trap and Negative Impacts

A primary finding was the "Reliance Trap." Novice programmers often deferred their own judgment to the system. If a progress bar turned green, students assumed correctness regardless of their program's actual output. The researchers categorized the fallout of faulty feedback into three types:

  • Impact on Programming Behavior (IPB): Retaining incorrect code due to false validation.
  • Impact on Time Spent (ITS): Wasting time "fixing" correct code due to false negatives.
  • Early Submission (ES): Submitting incomplete work after receiving premature validation.

Notably, 40% of faulty feedback instances resulted in a documented negative impact. False positives were particularly detrimental, with a 78.5% impact ratio.

Discussion and Future Directions

While the feedback system improved overall persistence—with 93% of students completing the task—it also introduced "lingering impacts" where early errors derailed students for significant periods.

Future development should move away from binary "correct/incorrect" indicators toward systems that communicate their own fallibility. For instance, a system might provide a confidence score (e.g., "70% certain this step is complete") to encourage continued student self-evaluation. Furthermore, transitioning from pattern-matching to semantics-based feedback could improve accuracy by understanding the underlying logic of student code rather than just its surface structure.

Summary updated

Abstract

Data-driven programming feedback systems can help novices to program in the absence of a human tutor. Prior evaluations showed that these systems improve learning in terms of test scores, or task completion efficiency. However, crucial aspects which can impact learning or reveal insights important for future improvement of such systems are ignored in these evaluations. These aspects include inherent fallibility of current state-of-the-art, students' programming behavior in response to correct/incorrect feedback, and effective/ineffective system components. Consequently, a great deal of knowledge is yet to be discovered about such systems. In this paper, we apply a multi-criteria evaluation with 5 criteria on a data-driven feedback system integrated within a block-based novice programming environment. Each criterion in the evaluation reveals a unique pivotal aspect of the system: 1) How accurate the feedback system is; 2) How it guides students throughout programming tasks; 3) How it helps students in task completion; 4) What happens when it goes wrong; and 5) How students respond generally to the system. Our evaluation results showed that the system was helpful to students due to its effective design and feedback representation despite being fallible. However, novices can be negatively impacted by this fallibility due to high reliance and lack of self-evaluation. The negative impacts include increased working time, implementation, or submission of incorrect/partially correct solutions. The evaluation results reinforced the necessity of multi-criteria system evaluations while revealing important insights helpful to ensuring proper usage of data-driven feedback systems, designing fallibility mitigation steps, and driving research for future improvement.
BibTeX
@misc{shabrina2022multicriteria,
  author = {Preya Shabrina and Samiha Marwan and Andrew Bennison and Min Chi and Thomas W. Price and Tiffany Barnes},
  title = {{A Multicriteria Evaluation for Data-Driven Programming Feedback Systems: Accuracy, Effectiveness, Fallibility, and Students' Response}},
  year = {2022},
  eprint = {2208.05326},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.05326},
}
Figure