"\"It's not like Jarvis✱

Ritvik Budhiraja, Ishika Joshi, Jagat Sesh Challa, Harshal D. Akolekar, Dhruv Kumar

TL;DR

This study explores how undergraduate computer science students in India are actually using ChatGPT for their coursework. It reveals that while students find ChatGPT highly beneficial for tasks like information gathering and coding assistance, they also face significant challenges with reliability and verification, prompting a call for more transparent and trustworthy AI tools in education.

What Problem Does This Paper Address?

The rapid rise of Large Language Models (LLMs) like ChatGPT has sparked widespread discussion about their potential impact on education. While much research has focused on the capabilities of LLMs from an instructor or researcher perspective (e.g., generating exercises, evaluating code), there's a notable gap in understanding the student experience. This paper aims to bridge that gap by investigating how undergraduate computer science students are actually using ChatGPT, what benefits they perceive, what challenges they encounter, and what improvements they suggest for its integration into their academic lives.

Methods

The researchers employed a mixed-methods approach, combining quantitative and qualitative data collection and analysis.

  • Surveys: A comprehensive online survey was distributed to 480 undergraduate computer science students across three Indian universities. The survey gathered data on usage frequency, specific use cases, perceived benefits and challenges, and overall confidence levels.
  • Interviews: Semi-structured interviews were conducted with 17 undergraduate computer science students from eight different universities. These interviews aimed to delve deeper into their experiences, motivations, habits, and perceptions regarding ChatGPT.
  • Data Analysis: Survey data was analyzed quantitatively, while interview transcripts underwent a three-layer thematic analysis (semantic coding, intermediate themes, latent coding) to identify overarching patterns and insights.

Datasets

The study did not introduce new datasets for training or evaluation. Instead, it collected data directly from undergraduate computer science students through surveys and interviews.

Key Insights and Ideas

  • Ubiquitous Adoption: The vast majority of surveyed students (84.8%) reported being "very familiar" with ChatGPT, with only 0.2% being unaware, indicating its widespread recognition within the CS student community.
  • Daily Drivers: A significant portion of students use ChatGPT daily (34%) or weekly (33.1%), highlighting its integration into their regular academic routines.
  • Coursework Powerhouse: Students primarily use ChatGPT for information gathering (89.2%), summarizing content (76.7%), and coding assistance (74.4%), demonstrating its utility in core CS tasks.
  • Confidence Boost: Over 57% of students expressed a high confidence level (4 or 5 out of 5) in ChatGPT's ability to aid with coursework-related queries.
  • The "Jarvis" Analogy: The title itself, "It's not like Jarvis, but it's pretty close!", captures the sentiment that while not perfect, ChatGPT offers a powerful, almost assistant-like experience for students.
  • The Double-Edged Sword: While appreciated for its speed and information access, students also expressed concerns about over-reliance potentially hindering deep learning and critical thinking.
  • Ethical Quandaries: Some students admitted to feeling guilt or viewing ChatGPT use as an "unethical shortcut," pointing to the ongoing debate around academic integrity.

Key Results

  • Usage Patterns: Students often start by exploring ChatGPT's general capabilities, then move to academic use cases like assignments and coding, eventually making it a regular tool.
  • Perceived Benefits: Key advantages include immediate feedback, access to more information, personalized learning, and on-demand tutoring.
  • Major Challenges: The most significant challenges reported are incorrect/unreliable answers (74.8%), the lack of a way to verify correctness (69.4%), and the inability to identify the sources of information (51.0%).
  • Overall Perception: A substantial majority (75.4%) have a positive or highly positive overall perception of ChatGPT as an educational tool.
  • Problem Frequency: While many students find ChatGPT helpful, 8.3% report running into response-related problems "very often," and 36% "often."

Limitations and Future Work

  • University-Specific Limitations: Due to resource and time constraints, the study was limited to a few universities, and the survey distribution was dependent on institutional approvals. This might introduce biases.
  • Snowball Sampling Bias: The use of snowball sampling for interviews could introduce biases related to the academic capacities and practices of the included universities.
  • Student-Led Interviews: While intended to foster trust, the fact that only student researchers conducted interviews might have led some participants to withhold certain information due to concerns about potential repercussions.
  • Exploratory Nature: The survey and interview questionnaires were not validated by prior literature, suggesting potential for refinement.

Future Work: The authors suggest developing LLM-based applications that provide confidence scores for answers, offer more transparent source attribution, and incorporate features for reporting and customizing responses. They also propose further research into the ethical implications and the development of domain-specific LLMs for computer science education.

Summary updated

BibTeX
@inproceedings{budhiraja2024it,
  author = {Ritvik Budhiraja and Ishika Joshi and Jagat Sesh Challa and Harshal D. Akolekar and Dhruv Kumar},
  title = {{"\textbackslash\{\}"It's not like Jarvis✱}},
  booktitle = {ACE},
  year = {2024},
  doi = {10.1145/3636243.3636257},
}
Figure