Immediate Data-Driven Positive Feedback Increases Engagement on Programming Homework for Novices

Samiha Marwan, Thomas W. Price, Min Chi, Tiffany BarnesCSEDM@EDM 2020

Introduction

Novice programming is often characterized by high uncertainty. Students frequently "tinker"—writing correct code only to delete it because they lack the confidence that they are on the right track. While Intelligent Tutoring Systems (ITS) typically focus on error detection and negative feedback, researchers at North Carolina State University have developed a system called DD-IPF (Data-Driven Immediate Positive Feedback) to leverage the power of positive reinforcement. By automatically recognizing sub-goal completion, the system provides real-time validation, significantly increasing student engagement.

The DD-IPF System

The DD-IPF system addresses the scalability challenge of manual rule-writing by using a data-driven approach to detect progress automatically.

Data-Driven Feature Detection

The system utilizes a Data-Driven Feature Detector (DDFD) that analyzes historical student solutions to identify "code shapes"—structural patterns indicative of successful progress. These raw features are then mapped to human-readable "Objectives" (e.g., "The length of the Squiral increases with each side").

User Interface

Integrated into the iSnap block-based programming environment, the system features a Progress Panel. As students work, objectives turn green upon completion. If a student inadvertently breaks a previously satisfied requirement, the objective turns red, providing an immediate signal to revert or fix the change. The system also triggers periodic pop-up messages to celebrate milestones or nudge students who have stalled.

The DD-IPF interface showing the Progress Panel with completed (green) and broken (red) objectives.

Experimental Design

The system was evaluated through a quasi-experimental study involving an introductory computer science course for non-majors. The control group (n=33) used the standard iSnap environment, while the experimental group (n=27) used the DD-IPF enhanced version. The task involved creating a "Squiral" (square spiral) drawing.

Results and Key Insights

The introduction of immediate positive feedback led to a dramatic shift in student behavior and performance.

Increased Engagement and Persistence

The most significant finding was the "Green Light" effect. Students in the experimental group spent more than double the time on the task compared to the control group (median 34.75 minutes vs. 13.54 minutes, $p < 0.01$). While control group students often submitted incomplete work quickly, experimental group students persisted until they "cleared" the progress panel.

Comparison of time spent (A) and scores (B) between the control and experimental groups.

Performance Gains

The experimental group achieved higher average project scores (86.4% vs. 76.9%). Although this result was marginally significant ($p = 0.19$), the medium effect size (Cohen’s $d = 0.45$) suggests a meaningful impact on learning outcomes. Furthermore, in the control group, 73% of students who finished faster than the median time submitted incorrect work; in the experimental group, almost no one submitted that early, suggesting the feedback prevented premature abandonment.

Trust and Hint Usage

Positive feedback appeared to build trust in the tutoring system. Students receiving DD-IPF used more hints and followed them more accurately. This suggests that by validating correct steps, the system established credibility, making students more receptive to guidance when they encountered difficulties.

Limitations and Future Work

Despite its success, the DD-IPF system faces challenges regarding algorithmic rigidity. Because it relies on historical data, it may fail to recognize novel or unconventional solutions that were not present in the training set. Future research will explore the use of Large Language Models (LLMs) to automate the labeling of data-driven features, potentially making the system easier to deploy across a wider variety of complex programming tasks.

Summary updated

BibTeX
@inproceedings{marwan2020immediate,
  author = {Samiha Marwan and Thomas W. Price and Min Chi and Tiffany Barnes},
  title = {{Immediate Data-Driven Positive Feedback Increases Engagement on Programming Homework for Novices}},
  booktitle = {CSEDM@EDM},
  year = {2020},
}
Figure