TL;DR
This paper investigates ChatGPT's performance in an introductory functional programming course, treating it as a student. The study found that ChatGPT, even without specific fine-tuning, achieved a respectable B- grade and ranked in the top half of the class, demonstrating surprising proficiency in complex coding tasks.
What Problem Does This Paper Address?
The rapid advancement and accessibility of large language models (LLMs) like ChatGPT have sparked significant interest in their potential impact on computer science education. While LLMs show promise in various tasks, including code generation, their actual performance on academic assignments and exams in a structured learning environment remained largely unquantified. This paper addresses the question of how well ChatGPT can perform in a real-world introductory functional programming course, evaluating its capabilities in understanding and solving programming problems, and exploring how prompt engineering can enhance its performance.
Methods
The researchers designed a two-pronged experimental approach:
- Unassisted Experiments: ChatGPT was tasked with completing all homework assignments and exams from a 2022 fall semester introductory functional programming course (OCaml) without any prior specific training or prompt engineering.
- Assisted Experiments: This phase involved "prompt engineering," where researchers guided ChatGPT using techniques similar to how a teaching assistant might help a student. These techniques included:
- Paraphrasing the problem description.
- Providing natural language hints.
- Teaching by example.
- Providing test cases.
ChatGPT's responses were automatically graded using the course's platform, and then manually graded for exams, using the same rubrics as human students. The study also compared ChatGPT's performance to the actual grade distributions of students in the course.
Datasets
The study used the homework assignments and exam questions from a specific introductory functional programming course (OCaml) offered in the Fall 2022 semester. The researchers specifically chose data from before ChatGPT's training cutoff (pre-2021) to ensure the model hadn't seen the exact questions before.
Key Insights and Ideas
- Surprising Proficiency: ChatGPT demonstrated a remarkable ability to solve many programming problems, even those involving complex concepts like higher-order functions and continuations, which are typically challenging for students.
- Struggles with Abstraction: Despite its coding prowess, ChatGPT struggled with understanding type specifications and inferring types of expressions, indicating a potential weakness in abstract reasoning.
- Prompt Engineering Matters: The "assisted" experiments, using prompt engineering, significantly improved ChatGPT's performance, highlighting the crucial role of effective prompting in leveraging LLMs.
- Potential as a Learning Tool: The study suggests ChatGPT can be a valuable tool for both students (offering real-time support and explanations) and instructors (assisting in generating assignments and providing feedback).
Key Results
- Overall Grade: ChatGPT achieved an overall grade of 67% (B-) in the course, ranking 155th out of 314 students in the unassisted mode.
- Homework Performance: In the unassisted mode, ChatGPT scored 100% on 16 out of 31 assignment problems.
- Assisted Improvement: Prompt engineering significantly boosted ChatGPT's performance. For instance, its rank improved from 220th to 155th overall. Assisted experiments showed improvements across homework, midterm, and final exams, with the final exam score increasing by 16.6 points.
- Exam Performance: ChatGPT excelled at code completion tasks but struggled with conceptual questions, requiring deeper understanding of programming paradigms.
- Error Analysis: Common errors included OCaml syntax errors, compilation errors (especially type mismatches), and logical errors.
Limitations and Future Work
- Model Version Dependency: The study's findings are based on a specific version of ChatGPT, and future updates could alter performance.
- Prompt Sensitivity: ChatGPT's output is highly dependent on the prompt, meaning results can vary even with minor prompt changes.
- Language Bias: Performance might differ across programming languages due to variations in their representation in training data.
- Reliability Concerns: The authors acknowledge that ChatGPT's reliability is not guaranteed, and its output requires careful evaluation.
- Future Work: The paper suggests further research into how ChatGPT can be integrated with other teaching methods to optimize its benefits while mitigating risks, and exploring its potential for generating personalized exercises and providing tailored feedback. The authors also emphasize the continued importance of programming education for developing critical thinking and code evaluation skills.
Summary updated