International Comparative Study on PISA Mathematics Achievement Test Based on Cognitive Diagnostic Models

Xiaopeng Wu, Rongxiu Wu, Hua-Hua Chang, Qiping Kong, Yi ZhangFrontiers in Psychology 2020 · DOI

TL;DR

This study leverages Cognitive Diagnostic Models (CDMs) to analyze PISA 2012 mathematics data from 10 countries, moving beyond simple performance rankings to understand students' mastery of specific cognitive attributes. By identifying distinct learning trajectories, the research offers a nuanced view of how students in different nations acquire mathematical knowledge, highlighting country-specific strengths and weaknesses.

What Problem Does This Paper Address?

International large-scale assessments like PISA are crucial for benchmarking educational systems. However, traditional PISA analyses primarily focus on overall scores, providing limited insight into why students perform as they do. This leaves a gap in understanding the granular cognitive skills and knowledge points that contribute to success or failure. The problem is that simply knowing a country's ranking doesn't reveal the underlying learning processes or specific areas where students struggle or excel. This study aims to bridge this gap by using CDMs to dissect PISA mathematics performance into finer-grained cognitive attributes, thereby offering a more diagnostic and actionable understanding of international mathematics education.

Methods

The core of this research lies in the application of Cognitive Diagnostic Models (CDMs). The authors first established a "Q-matrix," which maps 12 selected PISA 2012 mathematics items to 11 specific cognitive attributes across three dimensions: content, process, and context. They then compared eight different CDM models to determine the best fit for their data, ultimately selecting the Linear Logistic Model (LLM) due to its superior fit. Using the LLM, they analyzed the responses of over 24,000 students from 10 countries to estimate students' mastery of each of the 11 attributes. This allowed them to analyze knowledge states (patterns of attribute mastery) and, crucially, to infer learning trajectories – the sequential paths students might take to master these attributes.

Datasets

The study utilized data from the Program for International Student Assessment (PISA) 2012 mathematics test. Specifically, they analyzed item responses from 24,512 students across 10 countries: Australia, Canada, China (including Hong Kong, Macau, and Shanghai), Finland, Japan, Russia, Saudi Arabia, Singapore, the United Kingdom, and the United States. The analysis focused on 12 specific mathematics test items that were common across these countries.

Key Insights and Ideas

  • Beyond Rankings: The study powerfully demonstrates that PISA results can offer much more than just country rankings. CDMs unlock a deeper understanding of the cognitive processes and knowledge structures underlying student performance.
  • Attribute-Level Diagnosis: By breaking down mathematics into 11 distinct attributes (e.g., "Quantity," "Mathematical Operation," "Personal Context"), the research provides a diagnostic lens to identify specific areas of strength and weakness for students within and across countries.
  • Learning Trajectories as Roadmaps: The identification of learning trajectories offers a valuable conceptualization of how students might progress through mastering mathematical concepts. This moves from a static view of knowledge to a dynamic one, suggesting potential pathways for instruction and intervention.
  • Country-Specific Patterns: The findings reveal that while some countries share similar learning trajectories (e.g., US and China), others exhibit unique or complex paths (e.g., Singapore). This highlights the importance of context-specific educational approaches.
  • Prerequisite Relationships: The analysis suggests that certain attributes act as prerequisites for others (e.g., "Data and Uncertainty" for other content attributes), providing a hierarchical understanding of knowledge acquisition.

Key Results

  • Model Selection: The Linear Logistic Model (LLM) was found to have the best fit for analyzing the PISA mathematics data among the eight models tested.
  • Attribute Mastery Differences: Significant differences were observed in the mastery of the 11 cognitive attributes across the 10 countries. For instance, China excelled in content attributes like "Quantity" and "Space and Shape," while Japan led in "Data and Uncertainty." Conversely, the United States and Saudi Arabia showed weaknesses in content attributes.
  • Process Attribute Performance: Students generally performed better in process attributes than content attributes, with China, Japan, and Singapore showing strong and balanced mastery.
  • Context Attribute Performance: China performed well across most context attributes, while Japan excelled in societal contexts. The occupational context (C2) appeared to be a relatively difficult attribute for many students.
  • Learning Trajectories:
    • Australia, Canada, the UK, and Russia shared similar main learning trajectories.
  • Finland and Japan also exhibited consistent main learning trajectories.
  • The US and China shared the same primary learning trajectory.
  • Singapore's learning trajectory was the most complex and diverse.
  • The US and Saudi Arabia had relatively simple learning trajectories.
  • The attribute N4 (Data and Uncertainty) was often identified as a prerequisite for other content attributes.

Limitations and Future Work

  • Limited Item Set: The study was constrained to only 12 PISA 2012 mathematics items due to data availability. Future research could analyze a larger and more diverse set of items.
  • Cross-Sectional Data: The study used cross-sectional data, meaning it captured a snapshot in time. This limits the ability to definitively establish causal relationships or track individual student progress over time. Longitudinal studies are suggested for a more robust understanding of learning trajectories.
  • Attribute Definition: The attributes were defined based on existing PISA coding. Future work could involve deeper mining to refine attribute definitions and explore finer granularities.
  • External Factors: While the study acknowledges that external factors (e.g., curriculum, tutoring) influence learning trajectories, it did not deeply integrate these into the analysis. Future research could explore the interplay between student achievement, cognitive attributes, and environmental factors.
  • Generalizability: The study focused on specific countries. Expanding the analysis to a broader range of countries would enhance generalizability.
  • Process and Context Trajectories: The authors noted that clear rules for process and context attribute learning trajectories were not found, limiting their analysis in these areas. Further methodological development is needed here.

Summary updated

Abstract

As one of the most influential international large-scale educational assessments, the Program for International Student Assessment (PISA) provides a valuable platform for the horizontal comparisons and references of international education. The cognitive diagnostic model, a newly generated evaluation theory, can integrate measurement goals into the cognitive process model through cognitive analysis, which provides a better understanding of the mastery of students of fine-grained knowledge points. On the basis of the mathematical measurement framework of PISA 2012, 11 attributes have been formed from three dimensions in this study. Twelve test items with item responses from 24,512 students from 10 countries participated in answering were selected, and the analyses were divided into several steps. First, the relationships between the 11 attributes and the 12 test items were classified to form a Q matrix. Second, the cognitive model of the PISA mathematics test was established. The liner logistic model (LLM) with better model fit was selected as the parameter evaluation model through model comparisons. By analyzing the knowledge states of these countries and the prerequisite relations among the attributes, this study explored the different learning trajectories of students in the content field. The result showed that students from Australia, Canada, the United Kingdom, and Russia shared similar main learning trajectories, while Finland and Japan were consistent with their main learning trajectories. The primary learning trajectories of the United States and China were the same. Furthermore, the learning trajectory for Singapore was the most complicated, as it showed a diverse learning process, whereas the trajectory in the United States and Saudi Arabia was relatively simple. This study concluded the differences of the mastery of students of the 11 cognitive attributes from the three dimensions of content, process, and context across the 10 countries, which provided a reference for further understanding of the PISA test results in other countries and shed some evidence for a deeper understanding of the strengths and weaknesses of mathematics education in various countries.
BibTeX
@inproceedings{wu2020international,
  author = {Xiaopeng Wu and Rongxiu Wu and Hua-Hua Chang and Qiping Kong and Yi Zhang},
  title = {{International Comparative Study on PISA Mathematics Achievement Test Based on Cognitive Diagnostic Models}},
  booktitle = {Frontiers in Psychology},
  year = {2020},
  doi = {10.3389/fpsyg.2020.02230},
}
Figure