Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming

Victor-Alexandru Padurean, Adish Singla

Abstract

Generative models have demonstrated human-level proficiency in various benchmarks across domains like programming, natural sciences, and general knowledge. Despite these promising results on competitive benchmarks, they still struggle with seemingly simple problem-solving tasks typically carried out by elementary-level students. How do state-of-the-art models perform on standardized programming-related tests designed to assess computational thinking and problem-solving skills at schools? In this paper, we curate a novel benchmark involving computational thinking tests grounded in elementary visual programming domains. Our initial results show that state-of-the-art models like GPT-4o and Llama3 barely match the performance of an average school student. To further boost the performance of these models, we fine-tune them using a novel synthetic data generation methodology. The key idea is to develop a comprehensive dataset using symbolic methods that capture different skill levels, ranging from recognition of visual elements to multi-choice quizzes to synthesis-style tasks. We showcase how various aspects of symbolic information in synthetic data help improve fine-tuned models' performance. We will release the full implementation and datasets to facilitate further research on enhancing computational thinking in generative models.

Introduction

While Large Language Models (LLMs) can ace professional coding interviews, they often stumble on logic puzzles designed for elementary schoolers. There is a strange paradox in AI right now: GPT-4 can write complex Python scripts and pass university-level exams, yet it often fails at seemingly "simple" tasks like counting colored blocks or planning a path through a basic maze—tasks that a 10-year-old can do with ease.

Current benchmarks (like HumanEval) focus heavily on syntax and library knowledge rather than core Computational Thinking (CT) skills: decomposition, pattern recognition, abstraction, and algorithmic thinking. This research seeks to quantify this gap by testing state-of-the-art models on standardized programming tests used in primary and secondary schools.

Methodology

The researchers addressed the CT gap through a three-pronged approach:

  1. Benchmark Curation: They adapted three pedagogically validated tests (HOC, ACE, and CT-TEST) into a standardized format for LLMs. These tests involve "visual programming" (e.g., Scratch or Blockly) where an avatar must navigate a grid.
  2. Symbolic Data Generation: To improve model performance, they used symbolic methods (code emulators) to generate a massive synthetic dataset. This dataset includes "fine-grained" tasks like locating an avatar, tracing its path, and predicting the outcome of a single line of code.
  3. LLAMACT Models: They fine-tuned Llama3-8B on this synthetic data. Crucially, they trained the models to generate explanations—step-by-step "traces" of the avatar's movement—before providing the final answer. This teaches the model to "think" through code execution spatially.

Benchmark and Synthetic Datasets

The study utilizes a mix of real-world educational tests and a massive new synthetic training set:

  • HOC (Hour of Code): 20 code-writing tasks from the popular Maze Challenge.
  • ACE (Analyzing-Evaluating-Creating): 21 multiple-choice questions testing different cognitive levels of programming.
  • CT-TEST: 24 multiple-choice questions focused on algorithmic logic.
  • Synthetic Dataset: A novel collection of ~111,000 samples covering solution synthesis, multiple-choice questions, and fine-grained skills like "Grid Synthesis."
Breakdown of the synthetic dataset used for fine-tuning, highlighting the variety of tasks from basic sensing to complex grid synthesis.

Results and Discussion

The evaluation revealed several key insights:

  • The "Kid-Level" Wall: State-of-the-art models like GPT-4o and Llama3 barely match the performance of an average school student on these tests and lag significantly behind the top 25% of students.
  • Code Generation vs. Code Reasoning: Models are surprisingly better at writing code (Synthesis) than predicting what a given piece of code will do (Analyzing). This suggests they often "hallucinate" code that looks correct but fails to solve specific spatial constraints.
  • Small Models, Big Logic: By fine-tuning a small 8B parameter model (Llama3) with symbolic traces, the authors created LLAMACT, which matches the performance of the much larger GPT-4o.
  • Spatial Reasoning Bottleneck: The primary reason models fail is a lack of "mental execution"—the ability to track an avatar's orientation and position in a 2D grid.
Performance comparison across benchmarks (ACE, CT-TEST, HoC) for state-of-the-art models and human students.

Key performance metrics include:

  • GPT-4o: Achieved an overall accuracy of 53%, roughly equivalent to an average Grade 3-7 student.
  • Llama3-8B (Base): Scored only 22.9%.
  • LLAMACT (Fine-tuned): Jumped to 53%, matching GPT-4o.
  • Human Comparison: Top-performing Grade 7 students score around 85%, indicating a significant gap remains.

Conclusion and Future Work

The study highlights that spatial reasoning is the weak link in current LLM computational thinking. While fine-tuning with symbolic traces significantly improves performance, models still struggle to maintain an accurate internal "mental map" during inference.

Future work should focus on fine-tuning multi-modal models to "see" grids directly and developing techniques where models can interact with symbolic tools (like code interpreters) in real-time. Scaling this benchmark to more complex data structures and algorithms remains a critical challenge for achieving human-level computational logic in AI.

Summary updated

Abstract

Generative models have demonstrated human-level proficiency in various benchmarks across domains like programming, natural sciences, and general knowledge. Despite these promising results on competitive benchmarks, they still struggle with seemingly simple problem-solving tasks typically carried out by elementary-level students. How do state-of-the-art models perform on standardized programming-related tests designed to assess computational thinking and problem-solving skills at schools? In this paper, we curate a novel benchmark involving computational thinking tests grounded in elementary visual programming domains. Our initial results show that state-of-the-art models like GPT-4o and Llama3 barely match the performance of an average school student. To further boost the performance of these models, we fine-tune them using a novel synthetic data generation methodology. The key idea is to develop a comprehensive dataset using symbolic methods that capture different skill levels, ranging from recognition of visual elements to multi-choice quizzes to synthesis-style tasks. We showcase how various aspects of symbolic information in synthetic data help improve fine-tuned models' performance. We will release the full implementation and datasets to facilitate further research on enhancing computational thinking in generative models.
BibTeX
@inproceedings{padurean2024benchmarking,
  author = {Victor-Alexandru Padurean and Adish Singla},
  title = {{Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=gXiefc96eH},
}
Figure