Examples and Tutored Problems: Adaptive Support Using Assistance Scores

Amir Shareghi Najar, Antonija Mitrovic, Bruce M. McLarenIJCAI 2015

Introduction

In educational psychology, the "worked-example effect" suggests that novices learn most effectively when presented with step-by-step solutions rather than unsupported problem-solving. However, as learners gain expertise, these full examples can become redundant or even detrimental—a phenomenon known as the "expertise reversal effect." Most Intelligent Tutoring Systems (ITS) employ fixed instructional sequences, such as alternating one example with one problem. This "one-size-fits-all" approach often leads to inefficiency, where some students receive excessive help while others struggle with too little. This paper explores an adaptive strategy within the SQL-Tutor system to provide dynamically tailored scaffolding based on real-time student performance.

Methodology: Measuring Mastery

The researchers implemented an adaptive mechanism driven by two primary metrics:

  1. Assistance Score ($T$): This metric quantifies the level of help a student requires. SQL-Tutor provides six levels of feedback, ranging from simple "Correct/Incorrect" flags to full solutions. Higher-level hints contribute more to the $T$ score, serving as a proxy for the student's lack of mastery.
  2. Cognitive Efficiency ($CE$): $CE$ is calculated by dividing Performance ($P$, derived from the Assistance Score) by Mental Effort ($R$, self-reported by the student on a 9-point scale).

Adaptive Logic

The system uses the $CE$ score from a previous task to determine the next "preparation task." If a student demonstrates high efficiency ($CE > 1$), the system skips the preparation and moves directly to the next problem. For lower scores, the system provides varying levels of "faded" examples—partially completed solutions where the student must fill in specific SQL clauses.

The adaptive logic flowchart based on Cognitive Efficiency (CE) thresholds.

The "fading" process is targeted; the system identifies the specific SQL constraints the student struggled with and requires them to complete those specific parts of the query in the faded example. This ensures that the scaffolding is not only adjusted in quantity but also in content.

Experimental Design

The study involved 46 undergraduate students divided into a control group ($n=22$) and an experimental group ($n=24$). The control group followed a fixed sequence of alternating examples and problems, while the experimental group received the adaptive strategy based on their $CE$ scores.

Comparison of the experimental design for the control and experimental groups across 10 task pairs.

Key Results

The adaptive intervention yielded significant improvements in both learning outcomes and efficiency:

  • Higher Learning Gains: The experimental group achieved a normalized learning gain of 0.73, significantly higher than the control group's 0.56 ($p = 0.01$).
  • Reduced Time on Task: Students in the adaptive group completed the intervention in approximately 59 minutes, compared to 74 minutes for the control group ($p < 0.01$).
  • Post-test Performance: The adaptive group scored an average of 85.7% on the post-test, while the control group averaged 77.8%.
  • Effect Size: The Cohen’s $d$ for normalized learning gain was 0.75, indicating a medium-to-large effect size for the adaptive intervention.

Conclusion

By using assistance scores as a proxy for mastery, the researchers demonstrated that Intelligent Tutoring Systems can effectively bridge the gap between "watching" and "doing." The adaptive strategy not only improved procedural knowledge but also optimized the learning path, allowing advanced students to progress faster while providing necessary scaffolding for those who needed it. Future work will explore larger sample sizes and the integration of self-explanation metrics to further refine the student model.

Summary updated

BibTeX
@inproceedings{najar2015examples,
  author = {Amir Shareghi Najar and Antonija Mitrovic and Bruce M. McLaren},
  title = {{Examples and Tutored Problems: Adaptive Support Using Assistance Scores}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure