Outcome Over Learning? Evidence from Student Use of AI Code Assistants for Unit Test Writing

Daniel Hinbest, Jeremy S. Bradbury, Michael A. MiljanovicarXiv 2026 · arXiv

TL;DR

This study investigates how computer science students naturally interact with AI code assistants when tasked with writing unit tests. The researchers found that students overwhelmingly favor a "one-shot" approach—asking the AI to generate complete solutions from scratch—rather than using the tool for iterative learning or validation, potentially sacrificing long-term skill development for immediate task completion.

What Problem Does This Paper Address?

Generative AI (GenAI) has landed in computer science classrooms like a disruptive force. While educators hope students will use these tools as "24/7 personalized mentors," there is a growing concern that students are instead using them as "problem-solvers" to bypass the cognitive heavy lifting required for learning.

The authors specifically target unit testing, a critical software engineering skill that students often find difficult and unengaging. While previous research has looked at how AI affects performance, this paper pulls back the curtain on the process: what exactly are students typing into the chat box, and are they actually engaging with the output?

Methods

The researchers conducted a user study with 21 third-year undergraduate students enrolled in a Software Quality Assurance course. The study design involved:

  1. Baseline Survey: Assessing students' prior experience with AI and their confidence in writing unit tests.
  2. Unit Testing Tasks: Students completed four tasks in Python. Two tasks were performed with the help of Continue, an open-source AI assistant plugin for the PyCharm IDE (powered by GPT-4o or GPT-5).
  3. Log Analysis: The core of the research involved a manual qualitative analysis of the AI chat logs. The authors categorized interactions by:
  4. AI Mode: "Assistant" (chat-based) vs. "Agent" (direct code insertion).
  5. Interaction Style: "One-shot" (single prompt) vs. "Conversational" (iterative back-and-forth).
  6. Prompt Type: Requesting complete code, process help, validation, or debugging.

Datasets

The "dataset" in this study consists of the qualitative and quantitative data gathered from the 21 student participants, including:

  • Pre-study survey responses.
  • Completed Python test files.
  • Continue chat log files containing the exact prompts and AI responses.

The authors did not use a standard external benchmark; instead, they created two specific coding problems: Classify Triangle (focused on statement coverage) and Process Cart (focused on branch and loop coverage).

Key Insights and Ideas

  • The Confidence Gap: A striking finding was that students felt significantly more confident in their ability to use AI assistants than in their ability to write unit tests. This suggests that students may delegate tasks to AI not just for speed, but because they lack the foundational confidence to tackle the problem themselves.
  • The "Vending Machine" Approach: Most students treated the AI as a code generator rather than a collaborator. Between 76% and 79% of interactions were "one-shot," where the student provided the requirements and expected a finished product in return. Fig. 2: AI Interaction
  • Outcome Over Validation: Despite being in a Quality Assurance course, very few students asked the AI to validate the generated tests or check if they actually met the required code coverage. The focus was almost entirely on "Complete code" generation. Fig. 3: Prompt Type
  • Missing the "Aha!" Moment: By jumping straight to the solution, students bypassed the iterative process of debugging and refinement—the very stages where deep learning typically occurs in programming education.
Fig. 2: AI Interaction
Fig. 3: Prompt Type

Key Results

  • Prompt Dominance: In the first task, 90% of students asked the AI to generate "complete code." In the second task, this remained high at 84%.
  • Interaction Style: Only 21-24% of interactions were conversational. Most students provided the source code and instructions in a single prompt and accepted the first output.
  • Tool Usage: Students preferred the "Assistant" mode (57-68%) over the "Agent" mode, though many still used the assistant primarily for one-shot generation rather than asking questions. Fig. 1: AI Mode
  • Self-Reported vs. Actual Use: While only 14.3% of students claimed in the survey that they usually use AI for "full code generation," the log analysis showed that nearly all of them did exactly that during the study.
Fig. 1: AI Mode

Limitations and Future Work

The authors acknowledge several limitations:

  • Sample Size: The study only included 21 students from a single institution, which may not represent the broader student population.
  • Lack of Outcome Metrics: This paper focused on the process (the chat logs) but did not report on whether the AI-generated tests were actually correct or if the students' grades improved.
  • Selection Bias: Participation was voluntary and incentivized by a bonus mark, potentially attracting a specific type of student.

Future Work: The researchers plan to analyze the "non-AI" tasks to compare performance and explore how "accountability" (e.g., requiring students to submit their chat logs) might change behavior. They also suggest developing "learning-centric" AI assistants that provide hints and guidance rather than just spitting out finished code.

Summary updated

Abstract

Artificial intelligence (AI) code assistants are rapidly being adopted in programming education, yet little is known about how students naturally use these tools when given unrestricted access. In a user study investigating AI-assisted unit test writing, we observed that a majority of students adopted a one-shot interaction strategy, asking the AI to generate a complete unit test directly from the initial requirements. While this approach is capable of efficiently producing a solution, it shifts students' focus from learning how to write effective unit tests to simply completing the task. Moreover, because students rarely engaged in critically evaluating, refining, or iterating on AI-generated output, they may be missing opportunities to develop the critical thinking skills that are essential for effective and responsible use of AI. In an effort to shift student focus from outcome to learning we also discuss how training, accountability and improved learning task design can better support a more learning-centric responsible use of AI.
BibTeX
@misc{hinbest2026outcome,
  author = {Daniel Hinbest and Jeremy S. Bradbury and Michael A. Miljanovic},
  title = {{Outcome Over Learning? Evidence from Student Use of AI Code Assistants for Unit Test Writing}},
  year = {2026},
  eprint = {2611.14664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14664},
}
Figure