Can OpenAI o1 outperform humans in higher-order cognitive thinking?

Ehsan Latif, Yifan Zhou, Shuchen Guo, Lehong Shi, Yizhu Gao, Matthew Nyaaba, Arne Bewersdorff, Xiantong Yang, Xiaoming Zhai

TL;DR

This study benchmarks OpenAI’s o1-preview model against human experts across seven higher-order cognitive domains, ranging from scientific reasoning to creative thinking. While the AI significantly outperformed humans in structured tasks—achieving near-perfect scores in scientific literacy—it hit a surprising wall when faced with adaptive, ill-structured problem-solving.

What Problem Does This Paper Address?

As Large Language Models (LLMs) evolve, the conversation has shifted from "can they chat?" to "can they think?" Specifically, researchers are investigating whether AI can replicate higher-order thinking skills—the complex cognitive processes like systematic reasoning, critical evaluation, and creative synthesis that are essential for graduate-level academic and professional success.

Previous evaluations of LLMs have shown mixed results, particularly in areas requiring deep metacognition or nuanced scientific reasoning. This paper seeks to provide a rigorous, multi-domain comparison by testing OpenAI’s o1-preview against established human benchmarks used in educational research, identifying exactly where the "silicon ceiling" currently sits.

Methods

The researchers evaluated o1-preview across seven domains using a "human-in-the-loop" benchmarking approach. They utilized iterative prompting strategies, including zero-shot prompts and role-based prompts (e.g., "respond as a postgraduate student"), to elicit the model's best performance.

The AI's responses were then scored using the exact same rubrics applied to human cohorts in prior foundational studies. To ensure statistical rigor, the researchers conducted ten trials per domain and used one-sample t-tests and z-scores to compare the AI’s mean performance against human means (ranging from undergraduates to subject-matter experts).

Datasets

The study leveraged several gold-standard instruments for human cognitive assessment:

  • Critical Thinking: Ennis-Weir Critical Thinking Essay Test (EWCTET).
  • Systematic Thinking: The "Village of Abeesee" and "Lake Urmia Vignette" (LUV) instruments.
  • Computational Thinking: The CT Skills instrument and the Algorithmic Thinking Test for Adults (ATTA).
  • Data Literacy: Assessments developed by Merk et al. and Chen et al.
  • Creative Thinking: The Alternate Uses Task (AUT) for divergent thinking and the Remote Association Test (RAT) for convergent thinking.
  • Logical Reasoning: The LogiQA dataset, derived from the National Civil Servants Examination of China.
  • Scientific Reasoning: The Test of Scientific Literacy Skills (TOSLS).

Key Insights and Ideas

  • The "Structured Task" Superpower: The AI excels in environments with clear rules and predefined patterns. Its near-perfect performance in scientific reasoning (TOSLS) and data management suggests it can process structured information more reliably than even human experts.
  • Mastery of Complexity: In systematic thinking, the model was exceptionally adept at identifying feedback loops—a specific type of interconnected causal relationship that human students often struggle to map.
  • The Problem-Solving Paradox: In a striking contrast, while the model achieved a perfect score in algorithmic thinking, it failed significantly in the problem-solving sub-dimension of computational thinking (scoring 1.0 vs. a human mean of 3.68). This suggests that while the AI knows the "rules of the game," it struggles to apply them to novel, ill-structured real-world scenarios.
  • Creative Originality: Surprisingly, the AI outperformed humans in divergent thinking originality. It didn't just provide common answers; it generated novel uses for objects that were statistically more "original" than those produced by university students.

Key Results

The o1-preview model outperformed human benchmarks in five out of seven domains, as visualized in the performance overview below:

Performance overview of OpenAI o1-preview in higher-order thinking domains compared to human experts.
  • Scientific Reasoning: Achieved a near-perfect 0.99 (SD=0.12) on the TOSLS, surpassing biology experts who averaged 0.91.
  • Critical Thinking: Scored 24.33 on the EWCTET, significantly higher than the postgraduate mean of 18.39 (z = 0.90).
  • Systematic Thinking: On the Lake Urmia Vignette, it scored 46.1, more than doubling the human mean of 20.08 (z = 3.20).
  • Logical Reasoning: Reached 90% accuracy on LogiQA, edging out the human average of 86%.
  • Data Literacy: In the "Use Data" category, it scored 8.60 compared to a human post-test mean of 4.17 (z = 2.19).

Limitations and Future Work

The authors highlight several critical caveats:

  1. The "Ill-Structured" Gap: The model’s poor performance in adaptive problem-solving indicates that it is not yet a replacement for human intuition in messy, real-world contexts.
  2. Modality Bias: Because o1-preview is primarily text-based, visual components of tests (like interpreting complex graphs in the TOSLS) had to be converted to text, which may have simplified the cognitive load compared to what humans face.
  3. Assessment Design: The authors question if current benchmarks inadvertently favor the "pattern-matching" strengths of AI. They call for new, holistic assessments that better capture uniquely human traits like emotional intelligence, ethical reasoning, and metacognition.

Future research should focus on integrating multimodal inputs (vision and data) and longitudinal studies to see how AI-assisted education impacts long-term human cognitive development.

Summary updated

Abstract

This study evaluates the performance of OpenAI's o1-preview model in higher-order cognitive domains, including critical thinking, systematic thinking, computational thinking, data literacy, creative thinking, logical reasoning, and scientific reasoning. Using established benchmarks, we compared the o1-preview models's performance to human participants from diverse educational levels. o1-preview achieved a mean score of 24.33 on the Ennis-Weir Critical Thinking Essay Test (EWCTET), surpassing undergraduate (13.8) and postgraduate (18.39) participants (z = 1.60 and 0.90, respectively). In systematic thinking, it scored 46.1, SD = 4.12 on the Lake Urmia Vignette, significantly outperforming the human mean (20.08, SD = 8.13, z = 3.20). For data literacy, o1-preview scored 8.60, SD = 0.70 on Merk et al.'s "Use Data" dimension, compared to the human post-test mean of 4.17, SD = 2.02 (z = 2.19). On creative thinking tasks, the model achieved originality scores of 2.98, SD = 0.73, higher than the human mean of 1.74 (z = 0.71). In logical reasoning (LogiQA), it outperformed humans with average 90%, SD = 10% accuracy versus 86%, SD = 6.5% (z = 0.62). For scientific reasoning, it achieved near-perfect performance (mean = 0.99, SD = 0.12) on the TOSLS,, exceeding the highest human scores of 0.85, SD = 0.13 (z = 1.78). While o1-preview excelled in structured tasks, it showed limitations in problem-solving and adaptive reasoning. These results demonstrate the potential of AI to complement education in structured assessments but highlight the need for ethical oversight and refinement for broader applications.
BibTeX
@misc{latif2024can,
  author = {Ehsan Latif and Yifan Zhou and Shuchen Guo and Lehong Shi and Yizhu Gao and Matthew Nyaaba and Arne Bewersdorff and Xiantong Yang and Xiaoming Zhai},
  title = {{Can OpenAI o1 outperform humans in higher-order cognitive thinking?}},
  year = {2024},
  eprint = {2412.05753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.05753},
}
Figure