Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming

Ahana Ghosh, Liina Malva, Alkis Gotovos, Danial Hooshyar, Adish Singla

Abstract

We explore the role of quizzes in elementary visual programming domains popularly used for K-8 computing education. Prior work has studied various quiz types, such as fill-in-the-gap write-code questions. However, the overall impact of these quizzes is unclear: studies often show utility in the learning phase when enhanced with quizzes, though limited transfer of utility in the post-learning phase. In this paper, we aim to better understand the impact of different quiz types and whether quizzes focusing on diverse skills (e.g., code debugging and task design) would have higher utility. We design a study with Hour of Code: Maze Challenge by code.org as the base curriculum, interleaved with different quiz types. Specifically, we examine two learning groups: (i) HoC-ACE with diverse quizzes including solution tracing, code debugging, code equivalence, and task design; (ii) HoC-Fill with simple quizzes on solution finding. We conducted a large-scale study with 405 students in grades 6--7. Our results highlight that the curriculum enhanced with richer quizzes led to higher utility during the post-learning phase.

Introduction

In K-8 computer science education, platforms like Code.org and Scratch often use "microtasks" or quizzes to maintain student engagement. However, research on these quizzes presents a mixed picture: while they often improve performance during the immediate learning phase, they frequently fail to facilitate the transfer of knowledge to new or more complex problems in the post-learning phase.

Most existing curricula rely on simple "solution-finding" or "fill-in-the-gap" questions. This study investigates whether quizzes targeting higher-order cognitive skills—specifically analyzing, evaluating, and creating (the ACE framework)—can lead to superior long-term learning outcomes and skill transfer compared to traditional, simpler quiz formats.

Methodology and Study Design

The researchers conducted a large-scale study involving 405 students (grades 6–7) in Estonia. Using the Hour of Code (HoC): Maze Challenge as a baseline, they created two distinct learning paths:

  1. HoC-ACE (Experimental Group): Students encountered 21 diverse quizzes interleaved with 20 coding tasks. These quizzes were designed based on the ACE framework (Analyzing, Creating, Evaluating) and included tasks like code debugging, determining code equivalence, and "task design" (modifying the environment to fit a specific code snippet).
  2. HoC-Fill (Control Group): Students encountered the same 20 coding tasks but with 21 "fill-in-the-gap" quizzes focused purely on basic solution-finding.
Items for the HoC-ACE group, showing the sequence of tasks (T), diverse quizzes (Q), and the subsequent post-learning tasks (P).

The study was divided into a Learning Phase (completing the HoC curriculum with quizzes) and a Post-learning Phase, where students solved 15 advanced "PostHoC" tasks without any assistance.

The ACE Framework in Practice

The ACE quizzes were designed to move students beyond rote memorization. Instead of simply finding a path, students had to analyze existing code, evaluate its correctness, and sometimes design the task itself. This approach introduces "desirable difficulty," forcing deeper cognitive processing.

An example of a higher-order ACE quiz (Q13) requiring students to identify and fix errors in a code block to solve a grid-based maze.

Students in the ACE group spent significantly more time on quizzes (32 seconds vs. 18 seconds on average) and had lower first-attempt success rates, indicating that the pedagogical challenges were effectively engaging higher-order thinking.

Key Results and Insights

The study revealed a phenomenon termed the "Illusion of Learning." During the initial learning phase, both groups performed almost identically, with success rates around 88%. However, significant differences emerged during the post-learning transfer phase:

  • Superior Transfer: The HoC-ACE group significantly outperformed the HoC-Fill group ($\chi^2 = 75.1, p < 0.01$). The ACE group achieved a normalized score of 0.799 on advanced tasks, compared to 0.702 for the Fill group.
  • The Risk of Over-Scaffolding: Interestingly, students who used simple fill-in-the-gap quizzes (HoC-Fill) performed worse in the post-test than a baseline group with no quizzes at all. This suggests that "easy" quizzes might foster a superficial understanding that fails when scaffolding is removed.
  • Equalizing Effect: The benefits of the ACE curriculum were most pronounced in younger students and those with less prior programming experience, suggesting these quizzes are a powerful tool for novices.

Conclusion and Future Work

This research demonstrates that interleaving diverse, higher-order quizzes with standard coding exercises significantly improves a student's ability to solve new, complex programming problems. While simple quizzes help students progress through a lesson, only richer pedagogical challenges successfully facilitate knowledge transfer.

Future research could explore adaptive scaffolding, where quiz types change based on individual performance, and investigate whether these benefits extend to text-based programming languages like Python.

Summary updated

Abstract

We explore the role of quizzes in elementary visual programming domains popularly used for K-8 computing education. Prior work has studied various quiz types, such as fill-in-the-gap write-code questions. However, the overall impact of these quizzes is unclear: studies often show utility in the learning phase when enhanced with quizzes, though limited transfer of utility in the post-learning phase. In this paper, we aim to better understand the impact of different quiz types and whether quizzes focusing on diverse skills (e.g., code debugging and task design) would have higher utility. We design a study with Hour of Code: Maze Challenge by code.org as the base curriculum, interleaved with different quiz types. Specifically, we examine two learning groups: (i) HoC-ACE with diverse quizzes including solution tracing, code debugging, code equivalence, and task design; (ii) HoC-Fill with simple quizzes on solution finding. We conducted a large-scale study with 405 students in grades 6--7. Our results highlight that the curriculum enhanced with richer quizzes led to higher utility during the post-learning phase.
BibTeX
@inproceedings{ghosh2025exploring,
  author = {Ahana Ghosh and Liina Malva and Alkis Gotovos and Danial Hooshyar and Adish Singla},
  title = {{Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming}},
  booktitle = {56th ACM Technical Symposium on Computer Science Education (SIGCSE )},
  year = {2025},
  url = {https://openreview.net/forum?id=MT0nO6jzkz},
}
Figure