TL;DR
This paper investigates whether adding "distractors"—incorrect proof steps—to Proof Blocks (a drag-and-drop tool for learning mathematical proofs) improves student learning. While students using distractors showed the highest performance gains and spent significantly more time engaging with the material, the differences between groups were not statistically significant, suggesting that the "reading effect" of the study material may have overshadowed the specific impact of the tool's features.
What Problem Does This Paper Address?
Learning to write mathematical proofs is a notorious hurdle for undergraduate students. Proof Blocks were developed to scaffold this process: instead of staring at a blank page, students drag and drop pre-written lines of a proof into the correct logical order. This is the mathematical equivalent of "Parsons Problems" used in computer science education.
While previous research showed that Proof Blocks are as effective as writing proofs from scratch (and more efficient), it wasn't clear how to optimize them. Specifically, the authors wanted to know if including distractors—lines that look plausible but contain common logical errors—helps students learn better by forcing them to resolve "cognitive conflict." Does seeing a common mistake and learning why it's wrong lead to deeper understanding than simply seeing the correct steps?
Methods
The researchers conducted a randomized controlled trial (RCT) with 253 students from a Discrete Mathematics course at the University of Illinois Urbana-Champaign. Students were split into three groups:
- Off-topic (Control): Completed Proof Blocks on unrelated logic puzzles.
- No Distractors: Completed induction Proof Blocks with only the correct lines provided.
- Distractors: Completed the same induction Proof Blocks but with "paired" distractors—incorrect lines designed to mimic common student misconceptions (like failing to properly bound an inductive hypothesis).
The workflow followed a standard educational research design:
- Pretest: Two written induction proof problems.
- Instruction: All students read a textbook chapter on induction.
- Learning Activity: The assigned Proof Blocks activity with automated, line-based feedback.
- Posttest: An identical test to the pretest, administered one week later.
Datasets
The study utilized student performance data collected through OHEP (Open source Homework and Exam Platform).
- Participants: 201 students completed the full cycle (pretest, activity, and posttest).
- Content: The proofs focused on two areas: summation identities and closed forms for recursively defined functions.
- Evaluation: Proofs were graded using a validated 7-point rubric by a team of four researchers with high inter-rater reliability (Krippendorff’s alpha of 0.88).
Key Insights and Ideas
- The Power of Cognitive Conflict: The "Distractors" group was intentionally exposed to lines that represented common "traps." The theory is that by choosing a distractor and receiving targeted feedback, students resolve a misconception they might otherwise have carried into the exam.
- The "Reading Effect": Interestingly, all three groups (including the off-topic control) showed significant improvement from pretest to posttest. This suggests that simply reading the textbook chapter provided a massive baseline boost in learning, potentially making it harder to measure the incremental benefit of the Proof Blocks themselves.
- Engagement via Difficulty: Students in the Distractors group spent nearly double the time on the activity (19.3 minutes) compared to the No Distractors group (10.5 minutes). In educational psychology, more "time-on-task" is often a proxy for deeper processing, even if the immediate test results don't show a massive gap.
Key Results
- Learning Gains: The Distractors group saw a 35% score increase, the No Distractors group saw 32%, and the Off-topic group saw 25.7%.
- Statistical Significance: While the Distractors group performed best, the difference between the three groups was not statistically significant ($p = 0.114$ for distractors vs. control in the regression model).
- Predictive Power: A student's pretest score was the strongest predictor of their posttest score, accounting for much of the variance in the results.
- Time Investment: The time differences between all three groups were highly significant ($p < 0.001$), proving that distractors undeniably make the task more challenging and time-consuming.
Limitations and Future Work
The authors identify several reasons for the inconclusive results:
- Sample Size and Effect Size: The actual benefit of distractors might be small enough that a larger sample size is needed to reach statistical significance.
- Posttest Sensitivity: The written posttest might not have been sensitive enough to capture the specific nuances of what students learned by navigating distractors.
- The "Floor" of Learning: Because the textbook reading was so effective, it may have created a "ceiling effect" where the additional benefit of the Proof Blocks was marginalized.
Future Work: The researchers suggest tweaking the experimental design to give students longer, more complex learning activities or using a different topic where the "reading effect" isn't as dominant. They also aim to refine the distractors to ensure they are targeting the most "productive" misconceptions.
Summary updated