Employing a Think-Aloud Approach to Assess Young Learners' Computational Thinking

Yicheng PanProceedings of the 2019 AERA Annual Meeting 2019 · DOI

TL;DR

This paper explores using "think-aloud" protocols to peek inside the "black box" of how young learners develop Computational Thinking (CT). By asking students to verbalize their thought processes while interacting with and analyzing a Scratch game, researchers found that these interviews serve as both a powerful assessment tool and a metacognitive exercise that helps students articulate complex CS concepts they might struggle to implement solo.

What Problem Does This Paper Address?

As computer science education moves into primary and middle schools, educators face a significant hurdle: How do you measure a thought process?

Most current assessments for Computational Thinking (CT) rely on "artifacts" (the final code a student produces) or "block counting" (analyzing which Scratch blocks were used). However, these methods don't reveal why a student chose a specific logic or whether they actually understand the underlying principles. A student might copy-paste a script without understanding it, or conversely, have a brilliant algorithmic idea they simply lack the syntax to execute. The authors address this gap by testing a qualitative "think-aloud" approach to capture the nuances of abstraction, decomposition, and debugging in students aged 9–12.

Methods

The researchers conducted a study within an after-school "Scratch Club" involving 12 students (Grades 4-6). They developed a two-phase interview protocol centered around a Scratch program called Bee Maze:

  1. Phase I (Assessment as Learning): Students played the game for one minute and were asked to describe what the program does (Abstraction) and how they would build it from scratch (Problem Decomposition).
  2. Phase II (Assessment of/for Learning): Students were shown the actual underlying code blocks. They had to map specific code chunks to game behaviors (Code Reading) and suggest ways to extend or simplify the program (Creativity and Optimization).

The study utilized Brennan and Resnick’s (2012) CT Framework, which categorizes CT into Concepts (loops, events), Practices (debugging, iterating), and Perspectives (expressing, questioning).

Datasets

This was a qualitative study rather than a large-scale data-mining project. The "dataset" consisted of:

  • Audio-recorded and transcribed interviews from 12 participants (balanced by gender and representing various ethnic backgrounds).
  • The Bee Maze Scratch Program, a custom-built instrument designed to include specific CS concepts like parallelism, conditionals, and sensing.

Key Insights and Ideas

  • Prompting Scaffolds Decomposition: Students initially struggled to explain how to program the game. However, when researchers provided categories of Scratch blocks as prompts, students were significantly better at "decomposing" the problem into logical steps. This suggests that young learners often have the logic but need a "vocabulary" to express it.
  • The "Read-Write" Gap: There is a clear distinction between a student's ability to read code and write it. While 100% of students could correctly identify the code responsible for movement, and 92% identified the "Win" logic, they found the "Sensing" blocks (collision detection) much harder to explain or envision from scratch.
  • Assessment as a Learning Tool: The think-aloud process wasn't just a measurement; it acted as a metacognitive exercise. By explaining their thoughts, students were forced to reflect on their own understanding, effectively turning the assessment into a teaching moment.
  • Identifying "Pain Points": The protocol revealed that "Sensing" and "Collision Logic" are the primary hurdles for this age group. Nine out of twelve students identified avoiding obstacles as the most difficult part to program, highlighting where teachers should focus their instructional energy.

Key Results

The researchers quantified the students' ability to map code to function (Phase II):

  • 100% Success: Moving the sprite with arrow keys (Events/Parallelism).
  • 92% Success: Identifying the "Win" condition (Events).
  • 67% Success: Understanding the "Shaking" animation (Loops/Events), starting point resets, and obstacle avoidance (Conditionals).

Qualitatively, the "Creativity" prompts (Question 6) yielded sophisticated responses. Students suggested adding "lives," timers, multiple levels, and even "enemy AI" (e.g., a frog that chases the bee), demonstrating that they could conceptualize complex game mechanics even if they hadn't yet mastered the code to build them.

Limitations and Future Work

The primary limitation is the small sample size (n=12), which makes it difficult to generalize the findings to all young learners. Additionally, the study was conducted in a voluntary after-school setting, which might imply a higher-than-average baseline interest in CS among participants.

Future Work: The authors plan to bridge the gap between "talk" and "action" by examining students' verbal think-aloud data in direct conjunction with the computational artifacts (the actual code) they produce. The ultimate goal is to integrate these think-aloud protocols directly into classroom instruction as a standard way to support and assess CT development.

Summary updated

Abstract

Computational Thinking (CT) involves skills that help youth analyze and solve real-world problems drawing on computer science (CS) principles Programming is one way of promoting CT, though arguably CS is broader than programming and CT is even broader than CS (NRC, 2010). Nevertheless, programming and CT are closely related and with strong ties to CS In recent years, there has been increased attention in helping all students acquire CT knowledge and skills, particularly through programming initiatives. Yet, assessing or measuring CT development is a challenge, because assessments of CT remain underdeveloped and under-researched Recent efforts to address this challenge include the development of evaluative tools such as written tests or surveys (e.g., Litts, Kafai, Lui, Walker & Widman, 2017), analysis of students computational artifacts (Brennan & Resnick, 2012), interviews Further, given the widespread use of block-based programming environments among young learners (e.g., Scratch), some authors delineated assessments and rubrics for capturing students' learning of computational content by examining the number and use of Scratch blocks (e.g.,
BibTeX
@inproceedings{pan2019employing,
  author = {Yicheng Pan},
  title = {{Employing a Think-Aloud Approach to Assess Young Learners' Computational Thinking}},
  booktitle = {Proceedings of the 2019 AERA Annual Meeting},
  year = {2019},
  doi = {10.3102/1438757},
}
Figure