"It's not like Jarvis, but it's pretty close!" - Examining ChatGPT's Usage among Undergraduate Students in Computer Science

Ishika Joshi, Ritvik Budhiraja, Harshal D. Akolekar, Jagat Sesh Challa, Dhruv Kumar

TL;DR

This paper investigates how undergraduate computer science students in India are actually using ChatGPT for their coursework. The study reveals that while students find ChatGPT highly beneficial for tasks like information gathering and coding assistance, they also face significant challenges with its reliability and the need for effective prompt engineering. The findings offer valuable insights for educators and tool developers aiming to integrate LLMs into computer science education.

What Problem Does This Paper Address?

The rapid rise of Large Language Models (LLMs) like ChatGPT has sparked widespread discussion about their potential impact on education. While much existing research focuses on evaluating LLMs from an instructor's or researcher's perspective (e.g., their ability to generate programming exercises or solutions), there's a notable gap in understanding the actual, student-driven usage of these tools within academic contexts. This paper aims to bridge that gap by exploring how undergraduate computer science students perceive and utilize ChatGPT, identifying the benefits they experience, the challenges they encounter, and their suggestions for improvement.

Methods

The study employs a mixed-methods approach, combining quantitative surveys with qualitative interviews.

  • Surveys: A concise online survey was administered to 480 undergraduate computer science students across three Indian universities. The survey covered aspects like ChatGPT usage frequency, specific use cases, perceived advantages and disadvantages, confidence levels in using ChatGPT for coursework, and the frequency of encountering problems.
  • Interviews: Semi-structured interviews were conducted with 17 undergraduate computer science students from eight different universities. These interviews delved deeper into students' academic experiences with ChatGPT, exploring their motivations, habits, challenges, and overall perceptions.

Both survey responses and interview transcripts were analyzed using thematic analysis to identify recurring patterns and insights.

Datasets

This paper does not introduce new datasets or benchmarks. Instead, it collects data directly from undergraduate computer science students through surveys and interviews.

Key Insights and Ideas

  • Student-Centric Approach is Crucial: The study emphasizes the importance of understanding LLM usage from the student's perspective, moving beyond instructor-led evaluations.
  • ChatGPT as a "Productivity Booster": Students primarily leverage ChatGPT for tasks that save time and effort, such as quickly finding information, generating code snippets, and getting initial approaches to problems.
  • The "Prompt Engineering" Hurdle: Students often struggle to craft effective prompts to get accurate and useful responses, highlighting a significant learning curve.
  • Trust and Reliability Concerns: A major challenge is the unreliability and potential inaccuracy of ChatGPT's responses, leading to a need for constant verification.
  • Ethical Ambiguity: While students see ChatGPT as a powerful tool, there's an underlying concern about its potential to hinder deep learning and critical thinking, and instances of unethical use for academic dishonesty were noted.
  • Future of Education: Students perceive LLMs like ChatGPT as an inevitable and transformative force in education, akin to the advent of Google search.

Key Results

  • Positive Outlook: Over 57% of students expressed a positive to highly positive confidence level in using ChatGPT for coursework-related queries.
  • High Familiarity: 84.8% of respondents reported being "very familiar" with ChatGPT, with only 0.2% being unaware.
  • Frequent Usage: A significant portion of students use ChatGPT daily (34%) or weekly (33.1%).
  • Top Use Cases: The most frequent uses include gathering information (89.2%), summarizing learning content (76.7%), and assistance in coding (74.4%).
  • Perceived Advantages: Key benefits include immediate feedback (69.17%), access to more information (66.9%), and personalized learning (61.67%).
  • Major Challenges: The most cited challenges are incorrect/unreliable answers (74.8%), lack of verification methods (69.4%), and not knowing the sources of information (51.0%).
  • Problem Frequency: 45.4% of students sometimes encounter response-related problems, while 36% encounter them often.

Limitations and Future Work

Limitations: * The study was conducted with students from Indian universities, limiting generalizability to other cultural or educational contexts without further research. * The survey was only distributed to three universities due to approval constraints, and the interview participant recruitment relied on snowball sampling, which could introduce biases. * The research materials (surveys and questionnaires) were not validated by prior literature, being exploratory in nature.

Future Work: The authors suggest further research into:

  • Developing frameworks and theories that break down the learning process into steps to be incorporated into LLM-powered agents for programming assistance.
  • Leveraging LLMs' brainstorming capabilities to support student creativity while maintaining creative control.
  • Enhancing LLMs with features like confidence scores to indicate answer reliability and transparency regarding data sources.
  • Developing domain-specific LLM versions for various CS courses.
  • Creating training guides to educate students on effective and ethical prompt engineering techniques.
  • Conducting in-classroom and controlled experiments to scientifically evaluate the advantages and disadvantages of LLM-based applications.

Summary updated

BibTeX
@inproceedings{joshi2024it,
  author = {Ishika Joshi and Ritvik Budhiraja and Harshal D. Akolekar and Jagat Sesh Challa and Dhruv Kumar},
  title = {{"It's not like Jarvis, but it's pretty close!" - Examining ChatGPT's Usage among Undergraduate Students in Computer Science}},
  booktitle = {ACE},
  year = {2024},
  doi = {10.1145/3636243.3636257},
}
Figure