The impact of AI-assisted pair programming on student motivation, programming anxiety, collaborative learning, and programming performance: a comparative study with traditional pair programming and individual approaches

Gilbert Fan, Dandan Liu, Rui Zhang, Lihu PanInternational Journal of STEM Education 2025 · DOI

TL;DR

AI-assisted pair programming (AIPP) significantly outperforms individual programming and even traditional human-human pairing in terms of task performance and anxiety reduction. However, while AI is a highly effective "navigator" that boosts student confidence, it cannot yet replicate the deep social presence and collaborative richness provided by a human partner.

What Problem Does This Paper Address?

Learning to program is notoriously stressful. Students often face a "self-reinforcing cycle" where high cognitive demands lead to anxiety, which leads to avoidance, and ultimately, lower performance. Traditional pair programming (where two humans share one workstation) was designed to mitigate this, but it comes with its own friction: mismatched skill levels, scheduling headaches, and social friction.

With the rise of Large Language Models (LLMs), researchers are asking: Can an AI assistant serve as a viable "pair" for students? Does it actually help them learn and feel better about coding, or is it just a crutch that provides answers without engagement? This study specifically looks at the intersection of affective factors (motivation, anxiety) and objective performance to see if AI can bridge the gap.

Methods

The researchers employed a quasi-experimental design over two academic years (2023–2024) involving 234 undergraduate students in a Java web development course.

  • The Setup: Intact class sections were randomly assigned to one of three conditions:
  • AI-Assisted: Students paired with an LLM (GPT-3.5 Turbo in 2023; Claude 3 Opus in 2024).
  • Human-Human: Traditional pair programming.
  • Individual: Solo programming (the control group).
  • The Framework: The study used the Technology Acceptance Model (TAM) to see if "perceived usefulness" mediated the results.
  • Data Collection: Validated instruments (like the Intrinsic Motivation Inventory and Programming Anxiety Scale) were used at three time points (beginning, middle, and end of the semester) alongside objective metrics like code quality and error rates.

Datasets

This was a human-centric study rather than a model-training exercise. The "dataset" consisted of:

  • Participant Data: 234 undergraduates at Taiyuan University of Science and Technology.
  • Performance Metrics: Standardized assignment grades, code quality rubrics (functionality, readability, efficiency), and automated error rate analysis.
  • Survey Data: Longitudinal responses tracking motivation, anxiety, and social presence.

Key Insights and Ideas

  • The "Non-Judgmental" Advantage: One of the most striking findings was that AI-assisted groups saw a more substantial decrease in programming anxiety than human pairs. The authors suggest this is because AI is a "patient, non-judgmental" partner. Students feel safer experimenting and making "dumb" mistakes when the only witness is a chatbot.
  • Performance vs. Presence: AI-assisted students actually outperformed human pairs in programming tasks. However, they scored significantly lower on "social presence." This suggests that while AI is a superior technical tool for getting the job done, it doesn't yet fulfill the human need for "relatedness" or deep collaborative depth.
  • The Capability Plateau: Surprisingly, there were no significant differences in student outcomes between using GPT-3.5 Turbo and the more advanced Claude 3 Opus. This suggests that for undergraduate-level learning, we may have reached a "threshold of capability" where incremental improvements in the AI model don't necessarily translate to better educational outcomes.
  • Perception is Reality: Mediation analysis showed that the perceived usefulness of the AI was a major driver of success. If a student believes the AI is helping them, their intrinsic motivation rises and their anxiety drops, creating a positive feedback loop.

Key Results

  • Motivation & Anxiety: Compared to solo programmers, the AI group showed significantly higher intrinsic motivation ($p < .001, d = 0.35$) and lower anxiety ($p < .001$).
  • Task Performance: AI-assisted groups achieved higher scores in programming tasks than both individual and human-pair groups ($p < .001$).
  • Social Presence: Human-human pairs remained the gold standard for collaboration, scoring significantly higher on the Collaborative Learning Scale (CLS) and Social Presence Questionnaire (SPQ) ($p < .001$).
  • Model Fit: The structural equation model explained 42.3% of the variance in collaborative perceptions and 45.6% of the variance in programming performance.

Limitations and Future Work

The authors acknowledge several caveats:

  • Self-Reporting Bias: Much of the data on anxiety and motivation relies on students' own assessments.
  • The "Crutch" Risk: While performance was high, the study doesn't fully explore whether students are developing a long-term over-reliance on AI that might hinder their ability to solve complex problems independently in the future.
  • Context Specificity: The study was conducted within a specific cultural and academic context (Java students in China).
  • Rapid Evolution: By the time the paper was published, newer models (like GPT-4o or o1) had already been released.

Future Work: The researchers suggest exploring instructional strategies that combine the "best of both worlds"—leveraging AI for technical scaffolding while maintaining human-to-human interaction to ensure social and communication skills aren't lost.

Summary updated

Abstract

This study investigates the impact of AI-assisted pair programming on undergraduate students’ intrinsic motivation, programming anxiety, and performance, relative to both human–human pair programming and individual programming approaches. A quasi-experimental design was conducted over two academic years (2023–2024) with 234 undergraduate students in a Java web application development course. Intact class sections were randomly assigned to AI-assisted pair programming (using GPT-3.5 Turbo in 2023 and Claude 3 Opus in 2024), human–human pair programming, or individual programming conditions. Data on intrinsic motivation, programming anxiety, collaborative perceptions, and programming performance were collected at three time points using validated instruments. Compared to individual programming, AI-assisted pair programming significantly increased intrinsic motivation (p < .001, d = 0.35) and reduced programming anxiety (p < .001), producing outcomes comparable to human–human pair programming. AI-assisted groups also outperformed both individual and human–human groups in programming tasks (p < .001). However, human–human pair programming fostered the highest perceptions of collaboration and social presence, surpassing both AI-assisted and individual conditions (p < .001). Mediation analysis revealed that perceived usefulness of the AI assistant significantly mediated the relationship between the programming approach and student outcomes, highlighting the importance of positive perceptions in leveraging AI tools for educational benefits. No significant differences emerged between the two AI models employed, indicating that both GPT-3.5 Turbo and Claude 3 Opus provided similar benefits. While AI-assisted pair programming enhances motivation, reduces anxiety, and improves performance, it does not fully match the collaborative depth and social presence achieved through human–human pairing. These findings highlight the complementary strengths of AI and human interaction: AI support can bolster learning outcomes, yet human partners offer richer social engagement. As AI capabilities advance, educators should integrate such tools thoughtfully, ensuring that technology complements rather than replaces the interpersonal dynamics and skill development central to effective programming education.
BibTeX
@article{fan2025impact,
  author = {Gilbert Fan and Dandan Liu and Rui Zhang and Lihu Pan},
  title = {{The impact of AI-assisted pair programming on student motivation, programming anxiety, collaborative learning, and programming performance: a comparative study with traditional pair programming and individual approaches}},
  journal = {International Journal of STEM Education},
  year = {2025},
  doi = {10.1186/s40594-025-00537-3},
}
Figure