Abstract
Online learning platforms provide an opportunity to better understand students’ weaknesses by tracking both their learning behavior and knowledge. This information can then be used to assist teachers in making instructional decisions and to further guide those who are at risk of failure. In this paper, we tracked student learning data from a C++ programming course over a whole semester of their freshman year via the Trustie platform. A total of 17,854 code submissions were collected. We then used CppCheck, SonarQube and Trustie to capture the quality characteristics and submission characteristics of the code, including lineOfCode, cyclomaticComplexity, codeSmell, syntacticError, averageScore, submission, and logicError, and analyzed the impact of code quality on the assignment work results. Several factors were discovered that we believe can help teachers to develop more effective teaching strategies.
Introduction
Teaching introductory programming (CS1) is a complex challenge. While automated grading systems efficiently determine if a student's code produces the correct output, they often fail to capture the process—how a student arrived at a solution or the inherent quality of their code. By analyzing nearly 18,000 C++ code submissions from freshman students, this research explores the hidden links between submission behavior, code quality, and learning outcomes. The goal is to bridge the gap in understanding how fine-grained code quality metrics—such as cyclomatic complexity and "code smells"—correlate with submission patterns, helping instructors identify at-risk students and refine instructional design.
Methodology
The study tracked 120 first-year computer science students at the National University of Defense Technology (NUDT) over a full semester, covering 27 programming assignments.
Data Collection and Analysis
Students utilized the Trustie platform, an online programming environment that records every submission attempt, including those that fail to compile. This provided a granular view of the student's iterative development process.
To analyze code quality, the researchers employed a multi-stage automated pipeline:
- Feature Extraction: Tools like CppCheck and SonarQube were used to extract seven key features: lines of code (LOC), cyclomatic complexity, code smells, syntactic errors, logic errors, submission frequency, and average scores.
- Statistical Analysis: Given the non-normal distribution of the data, the team utilized Spearman’s rank correlation and regression analysis of residuals. This approach allowed them to isolate true relationships between variables by filtering out noise caused by high submission frequencies.
Results and Discussion
Submission Behavior and Workload
Analysis revealed that assignment difficulty had less impact on late submissions than the timing of the release. Late submissions peaked when multiple assignments were due in close proximity, suggesting that student "burnout" or poor time management is often a scheduling issue rather than a lack of technical ability.
The Evolution of Errors: From Syntax to Logic
A significant "Syntax-to-Logic Shift" was observed. Early in the semester, syntactic errors (grammar) are the primary roadblock. As students gain familiarity with C++ syntax, logic errors (incorrect problem-solving) become the dominant reason for failure. A high Spearman partial correlation of 0.93 was found between submission frequency and logic errors, confirming that logical misunderstandings are the main driver of "trial-and-error" submission behavior.
Code Quality and "Smells"
The research identified a moderate positive correlation (0.44) between "code smells" and logic errors. Students who write "smelly" code—such as using unnecessarily wide variable scopes or leaving unused variables—are significantly more likely to introduce logical bugs. The most frequent violations were variableScope and unusedVariable. While these don't prevent code from running, they correlate with higher failure rates. Furthermore, a correlation of 0.71 exists between Lines of Code (LOC) and cyclomatic complexity, indicating that verbose code in freshman assignments often leads to unnecessary complexity.
Conclusion
This study provides a data-driven roadmap for improving programming education. By monitoring code smells and logic errors rather than just compilation success, instructors can intervene more effectively. Future work will explore the impact of peer code reviews and seek to determine the optimal frequency for assignment releases to maximize learning without overwhelming students.
Summary updated