Measuring the Impact of Distractors on Student Learning Gains while Using Proof Blocks

Seth Poulsen, Hongxuan Chen, Yael Gertner, Benjamin Cosman, Matthew West, Geoffrey Herman

TL;DR

This paper investigates whether adding "distractors"—incorrect proof steps—to Proof Blocks (a drag-and-drop tool for learning mathematical proofs) improves student learning. While students using distractors showed the highest performance gains and spent significantly more time engaging with the material, the differences between groups were not statistically significant, suggesting that the "reading effect" of the study material may have overshadowed the specific impact of the tool's features.

What Problem Does This Paper Address?

Learning to write mathematical proofs is a notorious hurdle for undergraduate students. Proof Blocks were developed to scaffold this process: instead of staring at a blank page, students drag and drop pre-written lines of a proof into the correct logical order. This is the mathematical equivalent of "Parsons Problems" used in computer science education.

While previous research showed that Proof Blocks are as effective as writing proofs from scratch (and more efficient), it wasn't clear how to optimize them. Specifically, the authors wanted to know if including distractors—lines that look plausible but contain common logical errors—helps students learn better by forcing them to resolve "cognitive conflict." Does seeing a common mistake and learning why it's wrong lead to deeper understanding than simply seeing the correct steps?

Methods

The researchers conducted a randomized controlled trial (RCT) with 253 students from a Discrete Mathematics course at the University of Illinois Urbana-Champaign. Students were split into three groups:

  1. Off-topic (Control): Completed Proof Blocks on unrelated logic puzzles.
  2. No Distractors: Completed induction Proof Blocks with only the correct lines provided.
  3. Distractors: Completed the same induction Proof Blocks but with "paired" distractors—incorrect lines designed to mimic common student misconceptions (like failing to properly bound an inductive hypothesis).

The workflow followed a standard educational research design:

  • Pretest: Two written induction proof problems.
  • Instruction: All students read a textbook chapter on induction.
  • Learning Activity: The assigned Proof Blocks activity with automated, line-based feedback.
  • Posttest: An identical test to the pretest, administered one week later.

Datasets

The study utilized student performance data collected through OHEP (Open source Homework and Exam Platform).

  • Participants: 201 students completed the full cycle (pretest, activity, and posttest).
  • Content: The proofs focused on two areas: summation identities and closed forms for recursively defined functions.
  • Evaluation: Proofs were graded using a validated 7-point rubric by a team of four researchers with high inter-rater reliability (Krippendorff’s alpha of 0.88).

Key Insights and Ideas

  • The Power of Cognitive Conflict: The "Distractors" group was intentionally exposed to lines that represented common "traps." The theory is that by choosing a distractor and receiving targeted feedback, students resolve a misconception they might otherwise have carried into the exam.
  • The "Reading Effect": Interestingly, all three groups (including the off-topic control) showed significant improvement from pretest to posttest. This suggests that simply reading the textbook chapter provided a massive baseline boost in learning, potentially making it harder to measure the incremental benefit of the Proof Blocks themselves.
  • Engagement via Difficulty: Students in the Distractors group spent nearly double the time on the activity (19.3 minutes) compared to the No Distractors group (10.5 minutes). In educational psychology, more "time-on-task" is often a proxy for deeper processing, even if the immediate test results don't show a massive gap.

Key Results

  • Learning Gains: The Distractors group saw a 35% score increase, the No Distractors group saw 32%, and the Off-topic group saw 25.7%.
  • Statistical Significance: While the Distractors group performed best, the difference between the three groups was not statistically significant ($p = 0.114$ for distractors vs. control in the regression model).
  • Predictive Power: A student's pretest score was the strongest predictor of their posttest score, accounting for much of the variance in the results.
  • Time Investment: The time differences between all three groups were highly significant ($p < 0.001$), proving that distractors undeniably make the task more challenging and time-consuming.

Limitations and Future Work

The authors identify several reasons for the inconclusive results:

  • Sample Size and Effect Size: The actual benefit of distractors might be small enough that a larger sample size is needed to reach statistical significance.
  • Posttest Sensitivity: The written posttest might not have been sensitive enough to capture the specific nuances of what students learned by navigating distractors.
  • The "Floor" of Learning: Because the textbook reading was so effective, it may have created a "ceiling effect" where the additional benefit of the Proof Blocks was marginalized.

Future Work: The researchers suggest tweaking the experimental design to give students longer, more complex learning activities or using a different topic where the "reading effect" isn't as dominant. They also aim to refine the distractors to ensure they are targeting the most "productive" misconceptions.

Summary updated

Abstract

Background: Proof Blocks is a software tool that enables students to construct proofs by assembling prewritten lines and gives them automated feedback. Prior work on learning gains from Proof Blocks has focused on comparing learning gains from Proof Blocks against other learning activities such as writing proofs or reading. Purpose: The study described in this paper aims to compare learning gains from different variations of Proof Blocks. Specifically, we attempt to quantify the difference in learning gains for students who complete Proof Blocks problems with and without distractors. Methods: We conducted a randomized controlled trial with three experimental groups: a control group that completed an off-topic Proof Blocks activity, one that completed a \tool{} activity without distractors, and one that completed a Proof Blocks activity with distractors. All three groups read a book chapter on proof by induction before completing their activity. Findings: The group that completed the Proof Blocks activity with distractors performed better on the posttest than the group that completed the Proof Blocks without distractors, who in turn performed better than the group that completed the off-topic Proof Blocks activity. However, none of these differences were statistically significant. While the results of this study are inconclusive, we hope that it can serve as a foundation for future work.
BibTeX
@misc{poulsen2023measuring,
  author = {Seth Poulsen and Hongxuan Chen and Yael Gertner and Benjamin Cosman and Matthew West and Geoffrey Herman},
  title = {{Measuring the Impact of Distractors on Student Learning Gains while Using Proof Blocks}},
  year = {2023},
  eprint = {2311.00792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.00792},
}
Figure