TL;DR
This study explores the initial integration of Generative AI (GenAI) into an undergraduate engineering course, revealing that while it can enhance student performance and engagement, its use often remains at a foundational level of replacing existing tools or increasing efficiency. The findings highlight the importance of faculty experimentation and reflective practice in navigating this emerging technology.
What Problem Does This Paper Address?
The rapid emergence of Generative AI (GenAI) tools like ChatGPT and GitHub Copilot presents both opportunities and challenges for engineering education. While GenAI has the potential to revolutionize learning and teaching, there's a significant knowledge gap regarding how engineering faculty and teaching assistants (TAs) actually discover and implement these tools in their classrooms for the first time. Specifically, the paper addresses the question of how initial, exploratory integration of GenAI impacts student performance and what the experiences of educators are during this process.
Methods
The study employed an embedded descriptive single case study design. The research involved an engineering instructor (CI) and seven teaching assistants (TAs) in an introductory undergraduate computer systems course. Data collection included:
- Semi-structured Interviews: Conducted with the CI and TAs at the beginning, middle, and end of the academic semester to capture their evolving experiences, perceptions, and challenges.
- Formative Practice Exercises: Students' performance on these exercises was tracked to measure the impact of GenAI integration.
The study analyzed quantitative data from student performance and qualitative data from interviews using thematic analysis.
Datasets
Not discussed in the paper. The study focused on student performance data within the context of the course and qualitative data from interviews.
Key Insights and Ideas
- GenAI as a "Tool" at the Replacement Level: Initially, both the CI and TAs viewed GenAI primarily as another tool, akin to books or the internet, fitting Hughes' (2005) "replacement" level of technology integration. This means it was seen as a substitute for existing methods rather than a transformative element.
- Ambivalent Attitudes: Educators exhibited a mix of excitement and skepticism. While they recognized GenAI's potential for saving time, providing quick information, and acting as a tutor, concerns about accuracy, hallucination, privacy, and the potential for over-reliance were prevalent.
- Student Excitement, Waning Use: Students were initially surprised and excited to use GenAI in class, finding it helpful for tasks like checking code and confirming answers. However, their use of the technology tended to weaken over time.
- Focus on Efficiency, Not Transformation: GenAI integration primarily stayed at the level of replacing other instructional methods or increasing the efficiency of solving coding problems. The "amplification" level was observed, but "transformation" (fundamentally changing learning processes or content) was not yet evident.
- The Importance of "Learning by Doing": Both the CI and TAs emphasized that genuine learning, especially in engineering, still relies heavily on hands-on practice and learning from mistakes, with GenAI serving as a supplementary aid rather than a replacement for this core process.
- Prompt Engineering is Key: The quality of GenAI output was directly linked to the quality of prompts, highlighting the emerging importance of "prompt engineering" as a skill for both students and educators.
Key Results
- Improved Student Performance: Students' formative exercise performance generally increased after GenAI integration, with statistically significant improvements observed for most exercises. Effect sizes ranged from small to large, indicating substantial gains in post-exercise scores for several exercises.
- Initial Integration is Multidimensional: The study revealed that initial GenAI integration is complex, involving advantages, disadvantages, concerns, student reactions, teamwork dynamics, and pedagogical considerations.
- Faculty Skepticism and Evaluation: The CI and TAs engaged in a process of healthy skepticism, actively evaluating GenAI's contributions and limitations, which is crucial for informed adoption.
Limitations and Future Work
- Single Case Study: The study's reliance on a single introductory computer systems course limits the generalizability of its findings. Future research should explore GenAI integration in different engineering contexts and institutions, ideally over multiple years and sites.
- Limited Scope of GenAI Use: Students' use of GenAI was primarily focused on solving coding problems. Future research should investigate GenAI's application in other areas like design, development, and content creation.
- Instructor and TA Focus: The study primarily captured the perspectives of the instructor and TAs. Future research should include student perspectives to provide a more comprehensive understanding.
- Domain-Specific GenAI: General-purpose GenAI models may underperform in specialized engineering contexts. Research into developing domain-specific GenAI trained on engineering data is suggested.
- Longitudinal Impact: The study's duration was one semester. Longer-term studies are needed to understand the sustained impact of GenAI integration and how it evolves beyond initial novelty effects.
Summary updated