This academic paper, published in the Proceedings of the International Conference on Global Politics and Socio-Humanities, explores the pedagogical impact of Large Language Models (LLMs). Specifically, it proposes a comparative qualitative framework to evaluate how students in different academic tracks—Humanities/Social Sciences versus STEM—navigate the benefits and pitfalls of using ChatGPT.
TL;DR
The paper outlines a qualitative research design to compare the advantages and disadvantages of ChatGPT across different academic disciplines. Grounded in cognitivism, the study investigates how students use AI to manage cognitive load and the metacognitive processes involved in integrating AI-generated content into their education.
What Problem Does This Paper Address?
While the rapid adoption of ChatGPT in education is well-documented, the authors argue that there is a "notable gap" in understanding how its impact varies by field of study. STEM students might use AI for data analysis and coding, while Humanities students might use it for translation or essay drafting.
The paper seeks to move beyond general observations of "AI in the classroom" to pinpoint the specific "similarities and differences" in user experience between these two major academic branches. It also addresses the ethical tension students face: the desire for academic efficiency versus the risk of plagiarism and the loss of "human contact" in learning.
Methods
The authors employ a qualitative research design rooted in cognitivism theory. This theoretical lens focuses on internal mental structures and "metacognition" (thinking about one's own thinking).
The methodological approach consists of two phases:
- Open-ended Surveys: A 10-question questionnaire designed to categorize student experiences into "before" and "after" using ChatGPT.
- Semi-structured Interviews: In-depth, 30-minute individual sessions with a subset of participants to explore personalized experiences and emotional responses to AI tools.
- Thematic Analysis: The researchers use an iterative process to identify, code, and refine themes from the interview transcripts to draw comparative conclusions.
Datasets
The study does not use a pre-existing computational dataset. Instead, it generates a primary dataset through non-representative, purposeful sampling.
- Participants: 15–20 students.
- Distribution: An even split between high school and undergraduate students, and an even split between those studying Humanities/Social Sciences and STEM (8 participants per category).
- Geographic Scope: Students from two provinces in China.
Key Insights and Ideas
- AI as a Cognitive Load Reducer: The paper frames the decision to use ChatGPT as a metacognitive strategy. Students weigh the benefits of AI assistance against independent problem-solving to manage their "cognitive load," allowing them to focus on higher-level tasks.
- The Metacognitive "Paraphrase" Loop: A significant insight is the description of the student workflow: receiving AI output, experiencing "metacognitive apprehension" regarding plagiarism, and then intentionally paraphrasing or modifying the text to meet academic standards.
- Disciplinary Divergence: Preliminary literature review within the paper suggests that while ChatGPT shows "exceptional outcomes" in economics and critical thinking, its performance in specialized fields like medical education, law, and high-level mathematics is often "not totally sufficient."
- The Human Gap: A recurring theme is that generative models lack "full conceptual comprehension" and cannot replicate the nuanced interaction provided by human professors or tutors.
Key Results
As this paper focuses heavily on the research framework and preliminary qualitative design, the "results" are presented as the identified categories of impact:
- Advantages identified: Automatic essay grading, personalized feedback, and the ability to analyze large STEM datasets to identify patterns that traditional methods might miss.
- Disadvantages identified: Generation of imprecise/misleading information, lack of human-centric empathy, and ethical concerns regarding authorship and transparency.
- Comparative Goal: The study establishes that while the convenience of AI is a universal advantage, the risks (such as the "AI index" in plagiarism detectors like Turnitin) manifest differently depending on whether the student is submitting a creative essay or a technical report.
Limitations and Future Work
The authors candidly acknowledge several limitations:
- Self-Report Bias: Data relies on student memory and honesty, which can be influenced by social desirability bias.
- Timeliness: The study specifically looks at ChatGPT-3.5. Given the rapid release cycle of LLMs (like GPT-4 and beyond), the findings may have a short shelf-life.
- Generalizability: The small sample size (15-20 students) and specific geographic focus (China) mean the results may not apply to students in different cultural or educational systems.
- Future Directions: The authors suggest expanding this research to include the fields of arts and sports and conducting deeper investigations into the long-term societal impacts, such as potential increases in unemployment due to AI-driven automation.
Summary updated