Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming

Ahana Ghosh, Adish Singla, Alkis Gotovos, Danial Hooshyar, Liina Malva

TL;DR

This paper demonstrates that interleaving diverse, higher-order quizzes (focusing on debugging and task design) with standard coding exercises significantly improves a student's ability to solve new, complex programming problems. While simple "fill-in-the-gap" quizzes help students get through a lesson, only richer pedagogical challenges—aligned with higher levels of Bloom’s Taxonomy—successfully facilitate the transfer of knowledge to advanced tasks.

What Problem Does This Paper Address?

In K-8 computer science education, platforms like Code.org and Scratch often use "microtasks" or quizzes to keep students engaged. However, the research on these quizzes is a mixed bag: they often help students perform better during the lesson (the learning phase) but fail to help them solve different or harder problems later (the post-learning or transfer phase).

The authors identify a gap in how these quizzes are designed. Most existing curricula rely on simple "solution-finding" or "fill-in-the-gap" questions. This paper investigates whether quizzes that target higher-order cognitive skills—specifically analyzing, evaluating, and creating—can lead to better long-term learning outcomes and skill transfer compared to traditional, simpler quiz formats.

Methods

The researchers conducted a large-scale study with 405 students (grades 6–7) in Estonia. They used the popular Hour of Code (HoC): Maze Challenge as a baseline and created two distinct learning paths:

  1. HoC-ACE (Experimental Group): Students encountered 21 diverse quizzes interleaved with 20 coding tasks. These quizzes were designed based on the ACE framework (Analyzing, Creating, Evaluating) and included tasks like code debugging, determining code equivalence, and "task design" (modifying the environment to fit a specific code snippet).
  2. HoC-Fill (Control Group): Students encountered the same 20 coding tasks but with 21 "fill-in-the-gap" quizzes that focused purely on basic solution-finding.

The study was split into two phases: a Learning Phase (completing the HoC curriculum with quizzes) and a Post-learning Phase (solving 15 advanced "PostHoC" tasks without any quizzes or assistance).

Datasets

While this is a human-factors study rather than a machine learning paper, the "dataset" consists of the performance metrics of the 405 participating students across 17 different schools.

  • Base Curriculum: Hour of Code: Maze Challenge by Code.org.
  • Evaluation Benchmark: A custom-designed set of 15 tasks called PostHoC, which included both familiar tasks and "transfer tasks" that combined programming concepts in ways the students hadn't seen during the learning phase.

Key Insights and Ideas

  • The "Illusion of Learning": During the initial learning phase, both groups performed almost identically. If the researchers had stopped there, they might have concluded that quiz type doesn't matter. The real differences only emerged when students were asked to fly solo on harder problems.
  • Desirable Difficulty: Students in the ACE group spent significantly more time on their quizzes (32 seconds vs. 18 seconds on average) and had a lower first-attempt success rate. This suggests that the ACE quizzes provided a "desirable difficulty" that forced deeper processing of the material.
  • The Danger of Basic Quizzes: Surprisingly, the group that used simple fill-in-the-gap quizzes (HoC-Fill) actually performed worse in the post-test than a baseline group of students who had no quizzes at all. This suggests that over-reliance on "easy" quizzes might lead to a superficial understanding that collapses when the scaffolding is removed.
  • Bloom’s Taxonomy in Practice: By mapping quizzes to "Analyzing, Evaluating, and Creating," the authors successfully moved students beyond rote memorization toward a mental model of how and why code works.

Key Results

  • Learning Phase: No significant difference in success rates on coding tasks between the two groups (~88%).
  • Post-learning Phase (The Transfer): The HoC-ACE group significantly outperformed the HoC-Fill group ($\chi^2 = 75.1, p < 0.01$).
  • Aggregate Performance: The ACE group achieved a normalized score of 0.799 on advanced tasks, compared to 0.702 for the Fill group.
  • Impact of Prior Experience: The benefits of the ACE curriculum were most pronounced in younger students (Grade 6) and those with less prior programming experience, suggesting that high-quality interleaved quizzes are a powerful equalizer for novices.

Limitations and Future Work

  • Pre-test Absence: The authors did not use a pre-test to measure baseline knowledge to avoid student fatigue and "negative motivation," relying instead on school-level randomization and demographic surveys.
  • Fixed Sequencing: The study used a fixed order of tasks and quizzes. Future work could explore adaptive scaffolding, where the type of quiz changes based on a student's individual performance in real-time.
  • Long-term Retention: The post-test happened shortly after the learning phase. It remains to be seen if these benefits hold up over months rather than weeks.
  • Scaling to Text-based Code: The study focused on block-based visual programming. The researchers suggest extending this ACE framework to text-based languages like Python to see if the transfer effects are equally strong.

Summary updated

Abstract

We explore the role of quizzes in elementary visual programming domains popularly used for K-8 computing education. Prior work has studied various quiz types, such as fill-in-the-gap write-code questions. However, the overall impact of these quizzes is unclear: studies often show utility in the learning phase when enhanced with quizzes, though limited transfer of utility in the post-learning phase. In this paper, we aim to better understand the impact of different quiz types and whether quizzes focusing on diverse skills (e.g., code debugging and task design) would have higher utility. We design a study with Hour of Code: Maze Challenge by this http URL as the base curriculum, interleaved with different quiz types. Specifically, we examine two learning groups: (i) HoC-ACE with diverse quizzes including solution tracing, code debugging, code equivalence, and task design; (ii) HoC-Fill with simple quizzes on solution finding. We conducted a large-scale study with 405 students in grades 6--7. Our results highlight that the curriculum enhanced with richer quizzes led to higher utility during the post-learning phase.
BibTeX
@inproceedings{ghosh2025exploring,
  author = {Ahana Ghosh and Adish Singla and Alkis Gotovos and Danial Hooshyar and Liina Malva},
  title = {{Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming}},
  booktitle = {56th ACM Technical Symposium on Computer Science Education (SIGCSE )},
  year = {2025},
  url = {https://openreview.net/forum?id=MT0nO6jzkz},
}

Loading related content...

Figure