Abstract
Automated feedback systems in computer science education often rely on data-driven models to guide novice programmers. While these systems can provide valuable scaffolding, their inherent brittleness can lead to "incorrect" feedback signals. This study investigates student behavior in the presence of both accurate and inaccurate automated feedback within the iSnap block-based programming environment. Through a case-based analysis of 27 students completing a "Squiral" assignment, we find that while positive feedback helps struggling students overcome logical hurdles, false negatives can lead to significant time wasted on over-engineering, and false positives can cause students to submit non-functional code. The results highlight the "double-edged sword" of automated feedback and suggest a need for more transparent pedagogical interactions.
Introduction
Automated feedback systems, such as hints and progress trackers, are increasingly integrated into computer science curricula to support novice learners. Many of these systems are "data-driven," leveraging historical student logs to determine if a current student's progress aligns with successful past solutions. However, these detectors are often brittle; a student might produce a correct solution that deviates from historical patterns, leading to a False Negative (FN), or a broken solution might be erroneously validated as a False Positive (FP). This research explores the behavioral consequences of these incorrect signals: how do students respond when an automated system misleads them?
Methods
The researchers implemented the Data-Driven Positive Feedback (DDPF) system within iSnap, a block-based programming environment. The system utilizes a feature-detection algorithm to identify code block sequences representing "correct" logical properties based on prior student data.
Students interact with a "Progress Panel" containing four specific objectives for the "Squiral" assignment (creating a procedure to draw a spiraling square). When the system detects a completed objective, the corresponding label turns green. Conversely, if a subsequent edit "breaks" a previously completed objective, the label turns red.
The study analyzed log data from 27 students in an introductory computing course. Every code edit was recorded with a timestamp and the state of the automated detectors. The team employed a case-based approach, manually reviewing snapshots to compare automated detections against expert human judgment, tracking "Active Time" and "Idle Time" to measure engagement and frustration.
Key Insights and Behavioral Impacts
Feedback as a Logical Scaffold
For students who were significantly stuck, the DDPF system acted as a critical roadmap. Rather than just correcting syntax, the feedback guided students toward necessary logical structures, such as nested loops and variable increments. Case studies showed that students who initially failed the assignment without feedback were able to complete it within 14–29 minutes once the DDPF system was activated.
The High Cost of False Negatives
A significant finding was the detrimental impact of False Negatives. When the system failed to recognize a correct solution due to a non-standard coding style, students often prioritized the system's "green light" over their own functional code. For example, one student reached a 100% correct solution in just 2 minutes but spent an additional 12 minutes—five times the original effort—altering their code solely to satisfy the automated detector.
The "Green Light" Trap of False Positives
Conversely, False Positives created a "Green Light Trap." If the system erroneously marked an objective as complete, students frequently stopped working immediately, even if their program did not run correctly. These students relied more on the interface's validation than on testing their own visual output, leading to the submission of incomplete or non-functional work.
Results Summary
The study found that 25 out of 27 students eventually produced working solutions. Higher rates of early or correct detection were correlated with lower idle time, suggesting that positive reinforcement is a powerful motivator. However, incorrect detections in the refining phase significantly extended the time spent in the system without making actual progress. Students receiving False Positives were notably more likely to submit incomplete work, trusting the system's "all green" status over their own observations.
Discussion and Future Work
The findings underscore that automated feedback is a "double-edged sword." While it provides essential scaffolding for struggling students, its lack of transparency and potential for inaccuracy can lead to over-engineering or premature cessation of work.
The authors suggest that future systems should move beyond binary "Green/Red" signals. Incorporating "confidence levels" for detectors or prompting students to self-explain their code when the system is uncertain could foster a more collaborative pedagogical dialogue. Such improvements would help students develop their own expert judgment rather than over-relying on automated tools.
Summary updated