TL;DR
This paper introduces "class-bot," a novel system that uses "nudge theory" to guide undergraduate computer science students towards better software engineering practices. By providing automated, context-aware recommendations within GitHub issues, class-bot significantly improved students' code quality and productivity, offering a promising approach to combatting attrition in CS education.
What Problem Does This Paper Address?
The paper tackles the growing challenge of student retention in undergraduate computer science (CS) programs. Despite the increasing demand for software engineers, many students struggle with foundational programming courses, leading to high dropout rates, particularly among underrepresented groups. The authors identify "poor behaviors" like procrastination and neglecting software development processes as key contributors to these struggles. While automated tools are common for grading and feedback, they often lack the ability to proactively guide students toward better habits. This paper aims to bridge that gap by using automated nudges to improve student decision-making and encourage more robust software engineering practices.
Methods
The core innovation is the implementation of "developer recommendation choice architectures" within a system called "class-bot." This framework, inspired by nudge theory, influences user behavior by subtly altering the decision-making environment without outright rewards or restrictions. Class-bot works by automatically generating and updating GitHub issues on student project repositories. These issues are structured to guide students through the standard software development process (Requirements, Design, Implementation, Testing, Deployment) by highlighting relevant rubric items.
The system incorporates three key principles of choice architectures:
- Actionability: Class-bot automates the analysis of student repositories to determine task completion, reducing user effort.
- Feedback: It provides clear, visual feedback (checkmarks for completion, 'X' for unmet requirements) directly within the GitHub issues.
- Locality: Recommendations are placed directly within the student's project repository (spatial locality) and delivered via daily automated updates (temporal locality), making them readily accessible and timely.
The study compared the class-bot system against a baseline approach using an online rubric.
Datasets
Not discussed in the paper. The study evaluated student projects from an introductory Java programming course.
Key Insights and Ideas
- Nudge theory is applicable to software engineering education: The paper demonstrates that behavioral science principles can be effectively applied to improve how students approach programming assignments.
- Automated nudges can significantly impact student behavior: Class-bot's success suggests that well-designed automated systems can proactively guide students toward better practices, rather than just reacting to their code.
- Contextual and actionable feedback is crucial: Providing recommendations directly within the development environment (GitHub issues) and making them easy to understand and act upon is key to their effectiveness.
- Addressing "poor behaviors" is vital for retention: The findings highlight that issues like procrastination and skipping development phases are significant barriers to student success and retention in CS.
Key Results
The study found that students using class-bot demonstrated significant improvements in both project quality and productivity:
- Project Quality: Students using class-bot received significantly higher grades. While not statistically significant, projects without nudges incurred an average of 11 more points deducted.
- Student Productivity:
- The number of commits increased significantly.
- Code churn (lines added, deleted, or modified) increased significantly, suggesting more active engagement with the code.
- Students started assignments significantly earlier, with the average time to the first commit dropping from 8.32 days to 1.99 days.
- Projects with class-bot were submitted earlier on average.
Limitations and Future Work
Limitations: * The study was conducted in a single introductory Java programming course. * The paper acknowledges that students still sometimes waited until the end of the project to validate their work against expectations, indicating a need for more frequent self-validation. * Students desired more frequent updates from class-bot than the daily updates provided.
Future Work: * Develop more interactive mechanisms, such as checklists, to encourage more frequent student self-validation. * Implement more frequent notifications, potentially triggered by specific student actions (e.g., immediately after commits), to provide more persistent and timely recommendations. * Explore how to make automated recommendations less intrusive and more effective for professional software engineers, drawing lessons from this student-focused system.
Summary updated