Modeling Metacognitive and Cognitive Processes in Data Science Problem Solving (Student Abstract)

Maryam Alomair, Shimei Pan, Lujie Karen ChenAAAI 2023 · DOI

Abstract

Professional data science requires more than just technical proficiency; it demands strategic decision-making and metacognitive awareness. This study explores the use of "caselets"—miniature, real-world case studies—as a scalable method for teaching these complex problem-solving skills. By analyzing data from a graduate-level data science course (N=24), the research investigates the interplay between cognitive processes and metacognitive regulation. Using graph-based similarity analysis and predictive modeling, the study demonstrates that metacognitive awareness is a significant predictor of learning success. A Random Forest model achieved a Mean Absolute Percentage Error (MAPE) of 8.9% in predicting student learning gains, significantly outperforming linear baselines.

Introduction

Most data science (DS) curricula are biased toward teaching component skills—the what and the how of algorithms and tools. However, professional practice requires knowing when and why to apply specific methods to messy, real-world problems. While internships and long-term projects provide this experience, they are difficult to scale and often limit exposure to a single problem type.

The authors propose "caselets" as a solution for "deliberate practice." These structured, diverse, and time-efficient modules allow students to navigate strategic complexities. This research aims to model how students learn through these caselets, specifically focusing on the role of metacognition—the ability to monitor and regulate one's own thinking.

Methods and Data

The study involved 24 graduate students in a data science course featuring seven integrated caselets. Each caselet provided a problem context, data profile, and multiple-choice questions with explanatory feedback.

Analytical Framework

To understand the learning process, the researchers employed three primary techniques:

  1. Metacognitive Assessment: Students self-reported their "knowledge about regulation" (KC) and "regulation of cognition" (RC) using the Metacognitive Awareness Inventory (MAI).
  2. Correlation & Similarity Analysis: Graph-based representations visualized the relationships between cognitive and metacognitive processes.
  3. Predictive Modeling: Three regression models—Linear Regression (LR), Support Vector Machines (SVM), and Random Forests (RF)—were tested to predict learning gains.
Visualization of the relationships between metacognitive (RC, KC) and cognitive (C01-C07) processes within the learning framework.

Key Insights

The Role of Metacognition

The study confirms that metacognitive assessments are highly correlated with cognitive performance. As shown in the graph-based analysis, metacognitive regulation (RC) and knowledge (KC) are deeply intertwined with cognitive tasks. This suggests that teaching students to reflect on their problem-solving strategies is as vital as teaching the algorithms themselves.

Knowledge Component Clusters

Using cosine similarity, the researchers identified how different "Knowledge Components" (KCs) behave similarly based on student performance. The analysis revealed that "Model Diagnosis," "Data Understanding," and "Model Configuration" form a tight cluster.

Graph of Knowledge Component (KC) similarities, highlighting the tight coupling between Model Diagnosis, Data Understanding, and Model Configuration.

This clustering indicates that these skills are highly interdependent in the DS workflow; a student struggling with data understanding is likely to face difficulties in model configuration and diagnosis as well.

The Prior Knowledge Paradox

Interestingly, the authors found a negative relationship between pre-assessment scores and learning gain. Students with high initial scores showed lower gains, likely due to a "ceiling effect" where they had less "room to grow" within the specific constraints of the course modules.

Results

The Random Forest (RF) model was the most effective at predicting student learning gains, achieving a MAPE of 8.9%. This significantly outperformed the Linear Regression baseline (12.4% MAPE) and SVM, suggesting that the relationship between metacognition, prior knowledge, and learning gain is non-linear and complex. The most informative predictors for success were pre-assessment scores, caselet scores, and overall assignment performance.

Discussion and Future Work

While the small sample size (N=24) limits generalizability, this pilot study provides a foundation for more granular modeling of data science learning.

Future research will focus on:

  • Scaling: Developing an interactive environment to collect "trace data" from a larger pool of learners.
  • Granular Modeling: Moving from overall gain predictions to modeling growth at the individual knowledge component level to identify specific student bottlenecks.

Summary updated

BibTeX
@inproceedings{alomair2023modeling,
  author = {Maryam Alomair and Shimei Pan and Lujie Karen Chen},
  title = {{Modeling Metacognitive and Cognitive Processes in Data Science Problem Solving (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26936},
}
Figure