Student-AI Interaction: A Case Study of CS1 students

Matin Amoozadeh, Daye Nam, Daniel Prol, Ali Alfageeh, James Prather, Michael Hilton, Sruti Srinivasa Ragavan, Mohammad Amin AlipourKoli Calling 2024

TL;DR

This study investigates how novice computer science students (CS1) interact with generative AI (like ChatGPT) for programming tasks. It reveals that while AI can be a powerful aid, many students tend to over-rely on it, submitting full problem descriptions and accepting solutions without sufficient verification, potentially hindering their learning.

What Problem Does This Paper Address?

Generative AI tools like ChatGPT are rapidly being adopted in education, offering students instant explanations and code examples. However, this raises concerns about students becoming overly dependent, potentially hindering the development of fundamental programming skills and leading to academic dishonesty. While existing research explores AI's impact on productivity and code quality, there's a critical gap in understanding how novice programmers naturally interact with these tools without limitations, and how this impacts their learning and self-efficacy. This paper aims to fill that gap by studying CS1 students' unconstrained use of generative AI for programming tasks.

Methods

The researchers conducted a mixed-methods study with 15 CS1 students at a large US university. Participants used a custom VSCode plugin that integrated ChatGPT directly into their coding environment, allowing for natural interaction logging. They were tasked with completing three programming assignments. Data collection involved:

  • Automated logging: Capturing all ChatGPT prompts, responses, and participant interactions within the IDE.
  • Audio and video recordings: Documenting participants' screen activity, problem-solving behaviors, and reactions.
  • Pre- and post-study surveys: Assessing demographics, programming background, and student self-efficacy using Likert scales.

The analysis involved both qualitative coding of interactions and quantitative analysis of usage patterns and self-efficacy scores.

Datasets

Not discussed in the paper. The study used a custom VSCode plugin to collect data from participant interactions.

Key Insights and Ideas

  • The "AI Oracle" Phenomenon: A significant portion of students (about a third of the cases) directly submitted the full task description to ChatGPT, effectively asking the AI to solve the problem for them without prior independent effort. This "AI oracle" approach bypasses the learning process.
  • Verification is Lacking: Many students did not verify the AI-generated solutions, leading to incorrect submissions even when AI was used. This highlights a potential lack of critical evaluation skills.
  • Diverse Interaction Strategies: Students employed varied strategies, from "full description" (offloading the task) to "step-by-step" (breaking down problems) and "hybrid" (combining independent work with AI). The "full description" strategy, while seemingly efficient, often led to poorer outcomes.
  • Self-Efficacy is Complex: The impact of AI on self-efficacy was not uniform. While some students showed increased self-efficacy (especially those who used a more balanced approach), others experienced a decrease, particularly those who heavily relied on AI without understanding.
  • AI as a Tutor vs. Oracle: Generative AI can act as a helpful tutor for concepts and debugging, but it can also become an "oracle" that does the student's work, potentially hindering deep learning.

Key Results

  • Task Completion: Students successfully completed only 65% of the tasks where they used generative AI.
  • AI Usage Frequency: Participants used the generative AI plugin for 29 out of 40 completed submissions.
  • Prompting Strategies:
    • Full Description: Used in 17 instances across 9 participants, often early in the task.
  • Step-by-Step: Used by 4 participants, involving iterative problem-solving.
  • Hybrid/Partial: Used by remaining participants, tackling parts independently.
  • Information Needs: Prompts were categorized as seeking:
  • Entire Solution: 20 out of 60 prompts.
  • Coding Concepts: 19 out of 60 prompts.
  • Program Logic: 10 out of 60 prompts.
  • Debugging: 11 out of 60 prompts.
  • Response Exploitation: Participants either accepted the entire response (18/60 prompts), selectively used parts (23/60 prompts), or rejected and retried (19/60 prompts).
  • Self-Efficacy Change: One participant showed a significant increase in self-efficacy after using AI in a balanced way, while others who heavily relied on AI saw a decrease.

Limitations and Future Work

  • Replicability: The rapid evolution of generative AI means results might not be replicable as student behaviors change. Future work should involve replicating the study with different populations and AI tools.
  • Generalizability: The small sample size (n=15) limits the ability to generalize findings to all CS1 students. The study also relied on self-selection bias.
  • Novel Problem Types: The authors suggest developing new types of programming problems that are harder for current generative AI to solve directly, encouraging more genuine problem-solving and prompt engineering skills.
  • Pedagogical Support: Future research should focus on developing instructional strategies and AI tools that promote constructive AI use, self-regulation, and deeper understanding, rather than mere task completion.
  • Iterative Debugging: The study observed a lack of regular program execution and debugging, suggesting a need for tools and teaching methods that emphasize iterative development and error detection.

Summary updated

BibTeX
@inproceedings{amoozadeh2024student,
  author = {Matin Amoozadeh and Daye Nam and Daniel Prol and Ali Alfageeh and James Prather and Michael Hilton and Sruti Srinivasa Ragavan and Mohammad Amin Alipour},
  title = {{Student-AI Interaction: A Case Study of CS1 students}},
  booktitle = {Koli Calling},
  year = {2024},
}
Figure