Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming

Victor-Alexandru Padurean, Adish Singla

State-of-the-art AI models like GPT-4o and Llama 3 can ace complex Python coding benchmarks, yet they often struggle with "simple" visual programming puzzles designed for elementary schoolers. There is a striking contradiction in current AI capabilities: models can pass high-level university programming exams but fail at basic logic tasks—like stacking colored blocks or counting—that a 10-year-old can easily accomplish.

This paper introduces a new benchmark for Computational Thinking (CT) and demonstrates that fine-tuning smaller models with synthetically generated symbolic explanations can bridge this performance gap, bringing an 8B parameter model up to the level of GPT-4o.

The Problem with Current Benchmarks

The authors argue that existing benchmarks (like HumanEval) focus too much on syntax-heavy languages like Python and not enough on core computational thinking skills: decomposition, pattern recognition, abstraction, and algorithmic design. To address this, they curated a benchmark based on standardized school tests used in elementary visual programming (similar to Scratch or Blockly-style logic). They found that even the best models barely match the performance of an average school student on these fundamental tasks.

The Computational Thinking Benchmark

The researchers evaluated a suite of models on three established educational datasets:

  • HoC (Hour of Code): 20 code-writing (synthesis) tasks where the model must navigate an avatar through a maze using available logic blocks.
  • ACE: 21 multiple-choice questions covering the analysis, evaluation, and creation of code.
  • CT-Test: 24 multiple-choice questions focused on block-based programming logic.
Example of an Hour of Code (HoC) synthesis task

Methodology: Synthetic Data and Fine-Tuning

To improve model performance, the researchers didn't just scrape more web data. Instead, they used symbolic methods (programmatic emulators) to generate a massive synthetic dataset of over 586,000 tasks, selecting a high-quality subset of ~112,000 samples for training.

This synthetic data was broken down into fine-grained skills:

  • Basics: Simple spatial questions (e.g., "Where is the avatar?").
  • Tracing: Predicting the path of a code snippet step-by-step.
  • Grid Synthesis: Designing a map that fits a specific piece of code.

Crucially, they trained the models to produce symbolic explanations—step-by-step execution traces of the avatar moving through the grid—before outputting the final answer.

Key Insights and Results

The results highlight a significant gap between human students and current AI models, as well as the power of targeted fine-tuning.

Radar chart comparing AI models against student performance
  • The "Kids are Better" Gap: On the HoC synthesis tasks, an average Grade 7 student scores ~74%, while GPT-4o scores only ~30%. Top-tier students (top 25%) are nearly perfect (99.8%), leaving AI in the dust. Across all grades (3-7), the average student scores 61.2% overall, outperforming the best zero-shot AI models.
  • Symbolic Training Beats Scale: The base Llama-3-8B-Instruct model performed poorly, scoring only 22.9% overall and 0% on HoC synthesis tasks. However, when fine-tuned on high-quality symbolic data (creating the LLAMACT model), it reached an overall score of 53.0%, matching the performance of the much larger, multimodal GPT-4o. When provided with correct reasoning at inference, LLAMACT's score jumped to 72.2%.
  • The Power of the Trace: Models perform significantly better when forced to "trace" the execution (e.g., "the avatar moves to b2, then b3..."). This suggests that the failure of LLMs in computational thinking is often a failure of internal simulation and planning.
  • Reasoning vs. Synthesis: Interestingly, models are often better at picking the right answer in a multiple-choice format (ACE/CT-Test) than they are at explaining why that answer is correct or synthesizing a solution from scratch.

Limitations and Future Work

While fine-tuning with symbolic traces yielded massive improvements, challenges remain. Models still struggle with complex grid layouts and spatial reasoning, such as "missing wall" problems.

Furthermore, while the authors evaluated multimodal models like GPT-4o, they only fine-tuned text-based models. Fine-tuning the vision components of these models directly on visual programming grids represents a highly promising next step. Finally, the authors propose moving toward "interactive" reasoning, where a model can query a symbolic tool (like a code emulator) during its thought process to verify its logic in real-time.

Summary updated

Abstract

Generative models have demonstrated human-level proficiency in various benchmarks across domains like programming, natural sciences, and general knowledge. Despite these promising results on competitive benchmarks, they still struggle with seemingly simple problem-solving tasks typically carried out by elementary-level students. How do state-of-the-art models perform on standardized programming-related tests designed to assess computational thinking and problem-solving skills at schools? In this paper, we curate a novel benchmark involving computational thinking tests grounded in elementary visual programming domains. Our initial results show that state-of-the-art models like GPT-4o and Llama3 barely match the performance of an average school student. To further boost the performance of these models, we fine-tune them using a novel synthetic data generation methodology. The key idea is to develop a comprehensive dataset using symbolic methods that capture different skill levels, ranging from recognition of visual elements to multi-choice quizzes to synthesis-style tasks. We showcase how various aspects of symbolic information in synthetic data help improve fine-tuned models' performance. We will release the full implementation and datasets to facilitate further research on enhancing computational thinking in generative models.
BibTeX
@inproceedings{padurean2024benchmarking,
  author = {Victor-Alexandru Padurean and Adish Singla},
  title = {{Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=gXiefc96eH},
}
Figure