Improving Short and Long-term Learning in an Online Homework System

Ben Prystawski, Jacob Nogas, Andrew Petersen, Joseph Jay WilliamsCSEDM@EDM 2020

Introduction

In the rapidly evolving landscape of online education, instructors face a persistent pedagogical dilemma: when a student struggles with a concept, is it more effective to provide an immediate explanation of the correct solution or to require additional practice? While immediate feedback is a cornerstone of effective tutoring, there is a risk that explanations may serve as a temporary "crutch" rather than a foundation for deep learning. This study investigates this trade-off within the context of an introductory computer science course, specifically focusing on the acquisition of procedural skills like runtime analysis.

Theoretical Framework: The ICAP Model

The researchers framed their investigation using the ICAP framework (Interactive, Constructive, Active, Passive), which categorizes learning activities based on the level of cognitive engagement. Reading an explanation is generally classified as a "passive" or "active" activity (depending on how the student engages), whereas solving a problem is considered "constructive" or "interactive." While active learning is widely recognized as superior for long-term mastery, it is often more demanding and time-consuming for students. This study sought to determine if the immediate performance gains from passive explanations outweigh the long-term benefits of active problem-solving.

Methodology

The study was conducted using the Programming Course Resource System (PCRS) at the University of Toronto, involving 648 students enrolled in an introductory Python course. The experimental task required students to perform runtime complexity (Big-O) analysis on various for loops.

A sample of the multiple-choice options provided to students for runtime complexity analysis problems.

Experimental Design

The researchers employed a randomized factorial design with two primary factors:

  1. Explanation Factor: Students were randomly assigned to receive no explanation, a short rule-based explanation, or a long explanation featuring worked examples.
  2. Additional Problem Factor: Students were randomly assigned to either solve or skip one additional practice problem similar to the initial task.

To measure retention, the study tracked student performance across two timeframes. Short-term effects were measured by performance on four subsequent problems within the same homework session (Week 10). Long-term effects were assessed two weeks later (Week 12) using a post-test consisting of identical or slightly modified versions of the original problems.

The experimental mapping shows how initial Week 10 problems were paired with identical or modified versions in the Week 12 post-test to measure retention.

Results: The Trade-off Between Guidance and Practice

While the results did not reach the traditional $p < 0.05$ threshold for statistical significance, the data revealed consistent and suggestive trends regarding the efficacy of different interventions.

Short-term Gains from Explanations

In the immediate homework session, explanations appeared to provide a performance boost. Students who received explanations required slightly fewer attempts to reach the correct answer on subsequent problems ($p = 0.215$ for short explanations). This suggests that explanations act as a temporary formula that students can apply while the logic is still fresh in their working memory.

Long-term Retention through Practice

Conversely, the benefit of explanations largely evaporated over time. Two weeks later, the most significant predictor of success was whether the student had completed an additional practice problem during the initial session. Students in the "additional problem" group required fewer attempts to solve the post-test problems compared to those who did not receive extra practice ($p = 0.110$, improving to $p = 0.07$ when adjusting for variance). Interestingly, students took approximately the same number of attempts to solve identical problems two weeks later (2.07 vs. 2.02), indicating they were re-engaging with the logic rather than simply recalling a memorized answer.

Discussion: Crutches vs. Stickiness

The findings highlight a clear distinction between "performance" (immediate success) and "learning" (long-term retention). Explanations may improve immediate performance by providing a template for the task at hand, but they lack the "stickiness" required for long-term mastery. In contrast, the active effort required to solve an additional problem—while potentially more difficult in the moment—leads to deeper encoding of the procedural skill.

This aligns with the ICAP framework's emphasis on active engagement. The "passive" act of reading an explanation may help a student get through a homework set faster, but the "active" act of solving a problem is what builds lasting competence.

Conclusion

For designers of online homework systems, these results suggest that while explanations are helpful for maintaining student momentum, they should not replace active practice. A balanced approach might involve providing explanations to prevent frustration, followed by mandatory practice problems to ensure the concept is truly mastered. Future research should explore whether these effects vary based on student prior knowledge or the specific type of content being taught.

Summary updated

BibTeX
@inproceedings{prystawski2020improving,
  author = {Ben Prystawski and Jacob Nogas and Andrew Petersen and Joseph Jay Williams},
  title = {{Improving Short and Long-term Learning in an Online Homework System}},
  booktitle = {CSEDM@EDM},
  year = {2020},
}
Figure