Toward Predicting Success and Failure in CS2: A Mixed-Method Analysis

Yang Song, Lucas Layman, Yang Song 0019, Curry I. GuinnACM Southeast Regional Conference 2020 · DOI

TL;DR

This paper investigates the crucial transition from introductory programming (CS1) to data structures (CS2) by analyzing student grades and collecting qualitative feedback. They found that while CS1 grades are somewhat predictive of CS2 success, a significant portion of students see their grades drop, and specific CS1 concepts like linked lists and object-oriented programming are identified as major CS2 challenges.

What Problem Does This Paper Address?

The transition from a first programming course (CS1) to a second, more advanced course (CS2), often focusing on data structures, is a critical point for student retention in computer science. While CS1 success is widely studied, less is known about the factors that predict success or failure in CS2. This gap is problematic because failure in CS2 can be more impactful on a student's academic path. The authors aim to identify leading indicators of CS2 success and failure, understand student-perceived challenges in CS2, and assess how well CS1 prepares students for this next step. This knowledge can inform pedagogical improvements to help students navigate this transition more successfully.

Methods

The study employs a mixed-methods approach, combining quantitative analysis of student grades with qualitative data from student questionnaires.

  • Quantitative Analysis:
    • Grade Analysis: They analyzed over 2300 CS1 and CS2 course grades from Fall 2013 to Spring 2019 at the University of North Carolina Wilmington (UNCW). This involved mapping letter grades to a 4.0 scale and examining grade distributions, grade point changes between CS1 and CS2, and the relationship between CS1 and CS2 grades.
  • Concept Inventory: A modified version of the Second Computer Science 1 (SCS1) concept inventory was administered to 44 CS2 students in Spring 2019. This inventory assesses understanding of core CS1 concepts. The scores were then correlated with CS2 final grades.
  • Qualitative Analysis:
    • Student Questionnaire: 69 out of 79 CS2 students completed an exit questionnaire at the end of the Spring 2019 semester. This questionnaire asked students to rate CS1's preparation, identify helpful CS1 concepts, list CS1 concepts they wished they knew better, and describe the most challenging aspects of CS2. Responses were open-coded to identify recurring themes.

Datasets

  • Student Grades: A dataset of 2300 CS1 and CS2 course grades from Fall 2013 to Spring 2019 at UNCW. This includes 1641 CS1 grades and 706 CS2 grades.
  • Modified SCS1 Scores: Data from 44 CS2 students who took the modified SCS1 assessment in Spring 2019. 39 of these students completed the semester.
  • Student Questionnaire Responses: Responses from 69 CS2 students who completed the exit questionnaire in Spring 2019.

Key Insights and Ideas

  • The CS1-CS2 Grade Drop is Common: A significant majority of students (56%) experience a drop in their grade from CS1 to CS2, with 62% of these changes being within one grade point. This suggests that while not a drastic fall for most, CS2 is generally more challenging than CS1.
  • CS1 "C" Grade is a Warning Sign: Students who earn a "C" in CS1 are disproportionately likely to struggle in CS2, with 27% not achieving a "C" or better in CS2. This raises questions about whether a "C" in CS1 truly signifies readiness for CS2.
  • CS1 Concepts Matter, But Gaps Exist: While students found foundational concepts, iteration, and file manipulation from CS1 helpful in CS2, they identified "classes/objects" and "linked lists" as areas where they wished they had better preparation. This highlights a potential disconnect between CS1 curriculum and CS2's core data structure topics.
  • The Modified SCS1 Shows Promise: The modified SCS1 concept inventory demonstrated a statistically significant correlation with CS2 grades, suggesting it could be a useful early indicator of potential struggles in CS2. This is particularly interesting as the SCS1 is language-agnostic.
  • Python's Abstraction Might Mask Deeper Issues: The paper notes that Python's implicit handling of memory management and pointers might obscure concepts that are crucial for understanding data structures, potentially contributing to challenges with linked lists and object-oriented programming in CS2.

Key Results

  • Grade Trends: 56% of students had lower grades in CS2 than CS1, 18% improved, and 26% earned the same grade.
  • CS1 vs. CS2 Failure Rates: The outright failure rate in CS1 was 10.2%, while in CS2 it was 5.7%. However, the percentage of students not meeting the "C" threshold required for progression was 23.1% in CS1 and 15.0% in CS2.
  • SCS1 Correlation: A moderate, statistically significant correlation (p = 0.568, p < 0.001) was found between modified SCS1 scores and CS2 grade points earned.
  • Student Feedback:
    • 59% of students agreed that CS1 adequately prepared them for CS2.
  • The most helpful CS1 concepts were foundations (expressions, boolean values, syntax, vocabulary) and iteration.
  • The most challenging aspects of CS2 were identified as linked lists, classes/objects, and the pace of instruction.

Limitations and Future Work

  • Small Sample Size for SCS1: The correlation analysis for the modified SCS1 score was based on a small sample size (n=39), which the authors acknowledge and plan to address by replicating the analysis in future semesters.
  • Reliability of mSCS1: The reliability of the modified SCS1 assessment has not been formally evaluated, and its applicability outside the specific Python context is uncertain.
  • Instructor Bias: Student performance and questionnaire responses may be influenced by the specific pedagogical style of the instructor and course content.
  • Interrater Reliability: The qualitative coding of questionnaire responses was performed by a single author, limiting interrater reliability.
  • Anonymity: The questionnaire was not anonymous, which could have influenced student responses.

Future Work: The authors plan to continue investigating cross-course indicators of success, including more thorough analysis of the SCS1 inventory, the new Basic Data Structures Inventory (BDSI), in-process code execution metrics, and further qualitative feedback. They aim to improve student outcomes in CS2 by identifying and addressing the factors that contribute to failure.

Summary updated

BibTeX
@inproceedings{song2020toward,
  author = {Yang Song and Lucas Layman and Yang Song and Curry I. Guinn},
  title = {{Toward Predicting Success and Failure in CS2: A Mixed-Method Analysis}},
  booktitle = {ACM Southeast Regional Conference},
  year = {2020},
  doi = {10.1145/3374135.3385277},
}
Figure