Abstract
Professional data science requires more than just technical proficiency; it demands strategic decision-making and metacognitive awareness. This study explores the use of "caselets"—miniature, real-world case studies—as a scalable method for teaching these complex problem-solving skills. By analyzing data from a graduate-level data science course (N=24), the research investigates the interplay between cognitive processes and metacognitive regulation. Using graph-based similarity analysis and predictive modeling, the study demonstrates that metacognitive awareness is a significant predictor of learning success. A Random Forest model achieved a Mean Absolute Percentage Error (MAPE) of 8.9% in predicting student learning gains, significantly outperforming linear baselines.
Introduction
Most data science (DS) curricula are biased toward teaching component skills—the what and the how of algorithms and tools. However, professional practice requires knowing when and why to apply specific methods to messy, real-world problems. While internships and long-term projects provide this experience, they are difficult to scale and often limit exposure to a single problem type.
The authors propose "caselets" as a solution for "deliberate practice." These structured, diverse, and time-efficient modules allow students to navigate strategic complexities. This research aims to model how students learn through these caselets, specifically focusing on the role of metacognition—the ability to monitor and regulate one's own thinking.
Methods and Data
The study involved 24 graduate students in a data science course featuring seven integrated caselets. Each caselet provided a problem context, data profile, and multiple-choice questions with explanatory feedback.
Analytical Framework
To understand the learning process, the researchers employed three primary techniques:
- Metacognitive Assessment: Students self-reported their "knowledge about regulation" (KC) and "regulation of cognition" (RC) using the Metacognitive Awareness Inventory (MAI).
- Correlation & Similarity Analysis: Graph-based representations visualized the relationships between cognitive and metacognitive processes.
- Predictive Modeling: Three regression models—Linear Regression (LR), Support Vector Machines (SVM), and Random Forests (RF)—were tested to predict learning gains.
Key Insights
The Role of Metacognition
The study confirms that metacognitive assessments are highly correlated with cognitive performance. As shown in the graph-based analysis, metacognitive regulation (RC) and knowledge (KC) are deeply intertwined with cognitive tasks. This suggests that teaching students to reflect on their problem-solving strategies is as vital as teaching the algorithms themselves.
Knowledge Component Clusters
Using cosine similarity, the researchers identified how different "Knowledge Components" (KCs) behave similarly based on student performance. The analysis revealed that "Model Diagnosis," "Data Understanding," and "Model Configuration" form a tight cluster.
This clustering indicates that these skills are highly interdependent in the DS workflow; a student struggling with data understanding is likely to face difficulties in model configuration and diagnosis as well.
The Prior Knowledge Paradox
Interestingly, the authors found a negative relationship between pre-assessment scores and learning gain. Students with high initial scores showed lower gains, likely due to a "ceiling effect" where they had less "room to grow" within the specific constraints of the course modules.
Results
The Random Forest (RF) model was the most effective at predicting student learning gains, achieving a MAPE of 8.9%. This significantly outperformed the Linear Regression baseline (12.4% MAPE) and SVM, suggesting that the relationship between metacognition, prior knowledge, and learning gain is non-linear and complex. The most informative predictors for success were pre-assessment scores, caselet scores, and overall assignment performance.
Discussion and Future Work
While the small sample size (N=24) limits generalizability, this pilot study provides a foundation for more granular modeling of data science learning.
Future research will focus on:
- Scaling: Developing an interactive environment to collect "trace data" from a larger pool of learners.
- Granular Modeling: Moving from overall gain predictions to modeling growth at the individual knowledge component level to identify specific student bottlenecks.
Summary updated