Introduction
In the rapidly evolving landscape of online education, instructors face a persistent pedagogical dilemma: when a student struggles with a concept, is it more effective to provide an immediate explanation of the correct solution or to require additional practice? While immediate feedback is a cornerstone of effective tutoring, there is a risk that explanations may serve as a temporary "crutch" rather than a foundation for deep learning. This study investigates this trade-off within the context of an introductory computer science course, specifically focusing on the acquisition of procedural skills like runtime analysis.
Theoretical Framework: The ICAP Model
The researchers framed their investigation using the ICAP framework (Interactive, Constructive, Active, Passive), which categorizes learning activities based on the level of cognitive engagement. Reading an explanation is generally classified as a "passive" or "active" activity (depending on how the student engages), whereas solving a problem is considered "constructive" or "interactive." While active learning is widely recognized as superior for long-term mastery, it is often more demanding and time-consuming for students. This study sought to determine if the immediate performance gains from passive explanations outweigh the long-term benefits of active problem-solving.
Methodology
The study was conducted using the Programming Course Resource System (PCRS) at the University of Toronto, involving 648 students enrolled in an introductory Python course. The experimental task required students to perform runtime complexity (Big-O) analysis on various for loops.
Experimental Design
The researchers employed a randomized factorial design with two primary factors:
- Explanation Factor: Students were randomly assigned to receive no explanation, a short rule-based explanation, or a long explanation featuring worked examples.
- Additional Problem Factor: Students were randomly assigned to either solve or skip one additional practice problem similar to the initial task.
To measure retention, the study tracked student performance across two timeframes. Short-term effects were measured by performance on four subsequent problems within the same homework session (Week 10). Long-term effects were assessed two weeks later (Week 12) using a post-test consisting of identical or slightly modified versions of the original problems.
Results: The Trade-off Between Guidance and Practice
While the results did not reach the traditional $p < 0.05$ threshold for statistical significance, the data revealed consistent and suggestive trends regarding the efficacy of different interventions.
Short-term Gains from Explanations
In the immediate homework session, explanations appeared to provide a performance boost. Students who received explanations required slightly fewer attempts to reach the correct answer on subsequent problems ($p = 0.215$ for short explanations). This suggests that explanations act as a temporary formula that students can apply while the logic is still fresh in their working memory.
Long-term Retention through Practice
Conversely, the benefit of explanations largely evaporated over time. Two weeks later, the most significant predictor of success was whether the student had completed an additional practice problem during the initial session. Students in the "additional problem" group required fewer attempts to solve the post-test problems compared to those who did not receive extra practice ($p = 0.110$, improving to $p = 0.07$ when adjusting for variance). Interestingly, students took approximately the same number of attempts to solve identical problems two weeks later (2.07 vs. 2.02), indicating they were re-engaging with the logic rather than simply recalling a memorized answer.
Discussion: Crutches vs. Stickiness
The findings highlight a clear distinction between "performance" (immediate success) and "learning" (long-term retention). Explanations may improve immediate performance by providing a template for the task at hand, but they lack the "stickiness" required for long-term mastery. In contrast, the active effort required to solve an additional problem—while potentially more difficult in the moment—leads to deeper encoding of the procedural skill.
This aligns with the ICAP framework's emphasis on active engagement. The "passive" act of reading an explanation may help a student get through a homework set faster, but the "active" act of solving a problem is what builds lasting competence.
Conclusion
For designers of online homework systems, these results suggest that while explanations are helpful for maintaining student momentum, they should not replace active practice. A balanced approach might involve providing explanations to prevent frustration, followed by mandatory practice problems to ensure the concept is truly mastered. Future research should explore whether these effects vary based on student prior knowledge or the specific type of content being taught.
Summary updated