A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education

Ehsan Latif, Yifan Zhou, Shuchen Guo, Yizhu Gao, Lehong Shi, Matthew Nayaaba, Gyeong-Geon Lee, Liang Zhang, Arne Bewersdorff, Luyang Fang, Xiantong Yang, Huaqin Zhao, Hanqi Jiang, Haoran Lu, Jiaxi Li, Jichao Yu, Weihang You, Zhengliang Liu, Vincent Shung Liu, Hui Wang, Zihao Wu, Jin Lu, Fei Dou, Ping Ma, Ninghao Liu, Tianming Liu, Xiaoming ZhaiarXiv 2024 · DOI · OpenReview

TL;DR

This paper systematically assesses OpenAI's o1-preview model on 14 higher-order thinking skills crucial for education. The study reveals that o1-preview generally outperforms humans, especially in areas like systems thinking and computational thinking, but shows limitations in abstract reasoning, underscoring the need for human oversight and a recalibration of educational goals in an AI-integrated future.

What Problem Does This Paper Address?

The rapid advancement of AI, particularly large language models (LLMs) like OpenAI's o1-preview, presents a significant opportunity to revolutionize education and workforce development. However, there's a critical need to systematically evaluate AI's capabilities in performing complex, higher-order cognitive tasks that extend beyond basic knowledge retrieval. This paper addresses the gap in understanding how effectively AI can engage in and support skills like critical thinking, systems thinking, computational thinking, and creative problem-solving, which are vital for intellectual development and navigating complex real-world challenges. The authors aim to provide a comprehensive assessment of o1-preview's performance in these areas compared to human capabilities, informing how AI can be best integrated into educational settings.

Methods

The study employs a systematic assessment of OpenAI's o1-preview model across 14 distinct higher-order thinking dimensions. For each dimension, the researchers utilized existing, validated instruments with established human cognitive benchmarks. This involved adopting a "zero-shot" prompting strategy, where the model was given the problem and instructions in a single attempt without further feedback or refinement. The performance of o1-preview was then compared against human performance data from relevant literature or direct evaluations. The methodology focused on evaluating the model's ability to analyze, synthesize, reason, and problem-solve in complex, open-ended tasks.

Datasets

The paper utilized a variety of established datasets and instruments for evaluation, including:

  • Ennis-Weir Critical Thinking Essay Test: For assessing critical thinking.
  • Biological Systems Thinking Test, The Village of Abeesee, and The Lake Urmia Vignette (LUV): For assessing systems thinking.
  • Bebras Challenge problems and Algorithmic Thinking Test for Adults Instrument: For assessing computational thinking.
  • Leetcode coding questions: For assessing design thinking.
  • MATH dataset: For assessing metacognition.
  • Data literacy assessments from Merk et al. (2020) and Chen et al. (2023): For assessing data literacy.
  • Alternate Uses Task (AUT) and Remote Associates Test (RAT): For assessing creative thinking.
  • Raven's Progressive Matrices (RPMs): For assessing abstract reasoning.
  • FINQA dataset: For assessing quantitative reasoning.
  • LogiQA dataset: For assessing logical reasoning.
  • Analogical reasoning tasks from Musker et al. (2024): For assessing analogical reasoning.
  • Test of Scientific Literacy Skills (TOSLS): For assessing scientific reasoning.
  • Molecular dynamics simulations from the Protein Data Bank (PDB): For assessing spatial reasoning.

No new benchmarks were introduced; the focus was on using existing, validated instruments.

Key Insights and Ideas

  • AI as a Powerful Cognitive Augmenter: o1-preview demonstrates remarkable capabilities in many higher-order thinking domains, often exceeding human performance. This suggests AI can be a powerful tool to augment human learning and problem-solving.
  • The Nuance of "Higher-Order" Thinking: While AI excels in structured reasoning and pattern recognition, tasks requiring deep abstraction and nuanced interpretation (like abstract reasoning) still highlight human superiority, emphasizing the continued need for human oversight.
  • The Importance of Complementary Skills: The findings advocate for an educational paradigm shift, focusing on developing human skills that complement AI, such as creativity, critical thinking, and ethical reasoning, rather than solely on knowledge acquisition.
  • AI's Potential for Personalized Learning: The study suggests AI can offer more accessible feedback and personalized learning experiences, potentially transforming teaching methodologies and curricula.
  • The Evolving Landscape of AI in Education: The research underscores the transformative potential of AI in education and calls for a proactive recalibration of educational goals and practices to align with an AI-driven world.

Key Results

The study found that OpenAI's o1-preview model demonstrated significant capabilities across various higher-order thinking dimensions:

  • Outperformance: o1-preview achieved significantly better results (around 150% improvement) compared to humans in:
  • Systems Thinking
    • Computational Thinking
    • Data Literacy
    • Creative Thinking
    • Scientific Reasoning
    • Abstract Reasoning (though with caveats)
  • Underperformance: It slightly under-performed humans by about 25% in:
  • Logical Reasoning
    • Critical Thinking
    • Quantitative Reasoning
  • Comparable Performance: In Analogical Reasoning, both o1-preview and humans achieved perfect scores.
  • Specific Examples:
    • Critical Thinking: Achieved 81.25% on the Ennis-Weir Critical Thinking Essay Test, outperforming pre-intervention students and comparable to post-intervention improvements.
  • Systems Thinking: Outperformed middle school students by over 108.3% on the Biological Systems Thinking Test.
  • Computational Thinking: Achieved a mean score of 96.15% on the Bebras Challenge and Algorithmic Thinking Test, significantly outperforming undergraduate students.
  • Creative Thinking: Outperformed university students in both divergent and convergent thinking tasks.
  • Abstract Reasoning: Lagged behind psychology major students on Raven's Progressive Matrices.
  • Scientific Reasoning: Achieved a perfect score of 1.0 on the TOSLS, outperforming human participants.
  • Spatial Reasoning: Demonstrated high accuracy in molecular dynamics simulations.
  • Quantitative Reasoning: Exceeded general crowd performance but remained below human experts.
  • Metacognition: Showed impressive performance on intermediate algebra problems, exceeding typical university-level human performance.
  • Collaborative Thinking: Multi-agent setups using o1-preview outperformed single-agent setups and GPT-40 in math problem-solving and science group chats.

Limitations and Future Work

  • Limitations:
    • Abstract Reasoning: o1-preview showed limitations in abstract reasoning, where human psychology students outperformed it. This highlights the continued importance of human oversight for tasks requiring high-level abstraction.
  • Visual Data: The study noted that o1-preview's current inability to process visual data might have impacted its performance on tasks that rely heavily on visual representations, such as some abstract reasoning and data literacy tasks.
  • Limited Scope of Instruments: The evaluation was based on a select number of assessment instruments, and future studies should incorporate a wider range to provide a more comprehensive understanding.
  • Human-AI Interaction: Comparisons were primarily made with existing literature on human performance, and direct, real-time comparisons between AI and human performance in controlled educational settings are needed.
  • Potential Training Data Contamination: For scientific reasoning, the authors acknowledge the possibility that TOSLS items might have been included in the training data, potentially influencing the perfect score.
  • Future Work:
    • Multimodal Data Processing: Future investigation is needed to explore o1-preview's performance once its multimodal data processing feature is released.
  • Broader Assessment: Future studies should explore newer versions of the model and incorporate a wider range of higher-order thinking assessments, including those for creative thinking and ethical decision-making.
  • Direct Human-AI Interaction Studies: Conducting direct, real-time comparisons to better understand human-AI collaboration dynamics.
  • Domain-Specific Fine-tuning: Fine-tuning LLM-based agents for domain-specific tasks can further enhance their performance and specialization.
  • Addressing Limitations: Further development is needed in areas where o1-preview was found lacking, such as abstract reasoning and potentially in tasks requiring multimodal synthesis or high-end reasoning capabilities.

Summary updated

Abstract

As artificial intelligence (AI) continues to advance, it demonstrates capabilities comparable to human intelligence, with significant potential to transform education and workforce development. This study evaluates OpenAI o1-preview's ability to perform higher-order cognitive tasks across 14 dimensions, including critical thinking, systems thinking, computational thinking, design thinking, metacognition, data literacy, creative thinking, abstract reasoning, quantitative reasoning, logical reasoning, analogical reasoning, and scientific reasoning. We used validated instruments like the Ennis-Weir Critical Thinking Essay Test and the Biological Systems Thinking Test to compare the o1-preview's performance with human performance systematically. Our findings reveal that o1-preview outperforms humans in most categories, achieving 150% better results in systems thinking, computational thinking, data literacy, creative thinking, scientific reasoning, and abstract reasoning. However, compared to humans, it underperforms by around 25% in logical reasoning, critical thinking, and quantitative reasoning. In analogical reasoning, both o1-preview and humans achieved perfect scores. Despite these strengths, the o1-preview shows limitations in abstract reasoning, where human psychology students outperform it, highlighting the continued importance of human oversight in tasks requiring high-level abstraction. These results have significant educational implications, suggesting a shift toward developing human skills that complement AI, such as creativity, abstract reasoning, and critical thinking. This study emphasizes the transformative potential of AI in education and calls for a recalibration of educational goals, teaching methods, and curricula to align with an AI-driven world.
BibTeX
@misc{latif2024systematic,
  author = {Ehsan Latif and Yifan Zhou and Shuchen Guo and Yizhu Gao and Lehong Shi and Matthew Nayaaba and Gyeong-Geon Lee and Liang Zhang and Arne Bewersdorff and Luyang Fang and Xiantong Yang and Huaqin Zhao and Hanqi Jiang and Haoran Lu and Jiaxi Li and Jichao Yu and Weihang You and Zhengliang Liu and Vincent Shung Liu and Hui Wang and Zihao Wu and Jin Lu and Fei Dou and Ping Ma and Ninghao Liu and Tianming Liu and Xiaoming Zhai},
  title = {{A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education}},
  year = {2024},
  eprint = {2410.21287},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21287},
}
Figure