Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming

Paulo Blikstein, Marcelo Worsley, Chris Piech, Mehran Sahami, Steven H. Cooper, Daphne KollerJournal of the Learning Sciences 2014 · DOI

TL;DR

This paper introduces novel machine learning techniques to analyze student programming code snapshots, revealing distinct learning patterns and trajectories. While simple code update size didn't predict performance, changes in programming patterns over time showed a correlation with grades, suggesting a more nuanced understanding of student progress beyond final product assessment.

What Problem Does This Paper Address?

The paper tackles the challenge of understanding how students learn complex, open-ended tasks like computer programming, especially in the context of growing demand for scalable, student-centered learning. Traditional assessments often focus on the final product, missing the valuable insights hidden within the process of creation. The authors aim to bridge this gap by using learning analytics and machine learning to analyze high-frequency data (code snapshots) to identify patterns in student behavior and their relationship to learning outcomes. This is particularly relevant as computer science education expands, and the need for effective, scalable teaching and assessment methods becomes critical.

Methods

The authors employ a multi-stage approach using machine learning:

  • Code Update Differential: This initial metric quantifies changes between consecutive code snapshots, measuring lines added, deleted, or modified, and characters added, deleted, or modified. This provides a basic measure of code activity.
  • Clustering (k-means, x-means): Used to group students based on their code update patterns (size and frequency) across assignments and within assignments. This helps identify distinct programming styles or behaviors.
  • Dynamic Time Warping (DTW) and Scaled DTW: These techniques are used to compare code update "curves" (time series of code updates) between different assignments for the same student, even if the number of snapshots varies. This allows for the measurement of how much a student's programming pattern changes over time.
  • Hidden Markov Models (HMMs): In Study 2, HMMs are used to model student progression within a single assignment. They identify latent "states" (milestones in the programming process) and transitions between them, creating a state machine that represents typical student paths.
  • Similarity Metrics (Bag of Words, API Call Similarity, AST Change Severity): Used in Study 2 to compare code snapshots and build the HMM by quantifying the similarity between different program states.
  • Support Vector Machines (SVMs): Employed to combine different similarity metrics and predict expert labels for program similarity.

Datasets

The study utilizes a large dataset of code snapshots collected from:

  • 154,000 code snapshots from approximately 370 students enrolled in an introductory undergraduate programming methodology course.
  • Data was collected over four programming assignments across two academic quarters (spring and fall 2012).
  • Specific assignments used for analysis included "Checkerboard Karel" in Study 2.

Key Insights and Ideas

  • Programming is Pluralistic: The research supports the idea that there isn't a single "right" way to program. Students can achieve success through different approaches (tinkering vs. planning, or various developmental paths).
  • Process Over Product: Analyzing the process of programming (how code changes over time) offers richer insights than just looking at the final submitted code.
  • Change Matters: While the size of code updates didn't strongly predict performance, the amount of change in a student's overall programming pattern across assignments showed a correlation with grades. This suggests that students who adapt and evolve their approach tend to perform better.
  • Identifying "Sink States": The HMM analysis in Study 2 successfully identified "sink states" – unproductive or buggy states where students get stuck. This has potential for real-time intervention.
  • Generality of Patterns: The identified programming patterns and state machines were robust across different cohorts and even different instructors, suggesting these models have broad applicability.

Key Results

  • Experiments 1 & 2 (Code Update Size): No significant correlation was found between the average size of code updates and course performance (grades). Clustering revealed distinct programming styles, but these were weakly related to performance.
  • Experiment 3 (Code Update Pattern Change): A correlation was found between the amount of change in students' programming patterns over the course and their assignment and exam grades. Higher performing students showed more significant changes in their update patterns. While not always statistically strong (p < .08), post-hoc analysis supported these findings.
  • Study 2 (Karel Assignment):
    • Developed a state machine with 26 prototypical states for the Karel assignment.
  • Identified "sink states" where students struggled.
  • Found that trajectories with fewer sink states correlated with better student performance on exams.
  • The Alpha group (higher change) generally performed better than the Gamma group (lower change).
  • The distinction between Alpha and Gamma groups on midterm scores was significant (7.9% difference, p < .04).

Limitations and Future Work

  • Unit of Work Approximation: The study used save/compile events as a proxy for a "unit of work," which might not perfectly capture all student activity.
  • Off-Task Behavior: The current technology cannot track what students do when they are not actively programming or distinguish on-task from off-task time.
  • Human Labeling: Study 2 required laborious human labeling for similarity determination.
  • Sensitivity to Workflow Choices: The results could be sensitive to choices made in data normalization, time warping, and clustering.
  • Signal vs. Noise: The authors caution that large datasets don't automatically yield insights; distinguishing signal from noise requires careful design and human intervention.

Future work could extend these methods to other open-ended tasks (robotics, media creation), develop real-time interventions based on identified patterns, and create more sophisticated assessments that leverage student-generated data. The authors advocate for "programming pluralism," embracing diverse learning paths rather than enforcing monolithic approaches.

Summary updated

Abstract

New high-frequency, automated data collection and analysis algorithms could offer new insights into complex learning processes, especially for tasks in which students have opportunities to generate unique open-ended artifacts such as computer programs. These approaches should be particularly useful because the need for scalable project-based and student-centered learning is growing considerably. In this article, we present studies focused on how students learn computer programming, based on data drawn from 154,000 code snapshots of computer programs under development by approximately 370 students enrolled in an introductory undergraduate programming course. We use methods from machine learning to discover patterns in the data and try to predict final exam grades. We begin with a set of exploratory experiments that use fully automated techniques to investigate how much students change their programming behavior throughout all assignments in the course. The results show that students’ change in programming patterns is only weakly predictive of course performance. We subsequently hone in on 1 single assignment, trying to map students’ learning process and trajectories and automatically identify productive and unproductive (sink) states within these trajectories. Results show that our process-based metric has better predictive power for final exams than the midterm grades. We conclude with recommendations about the use of such methods for assessment, real-time feedback, and course improvement.
BibTeX
@article{blikstein2014programming,
  author = {Paulo Blikstein and Marcelo Worsley and Chris Piech and Mehran Sahami and Steven H. Cooper and Daphne Koller},
  title = {{Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming}},
  journal = {Journal of the Learning Sciences},
  year = {2014},
  doi = {10.1080/10508406.2014.954750},
}
Figure