Can OpenAI o1 outperform humans in higher-order cognitive thinking?

Ehsan Latif, Yifan Zhou, Shuchen Guo, Yizhu Gao, Lehong Shi, Matthew Nyaaba, Arne Bewerdorff, Xiantong Yang, Xiaoming Zhaı

Executive Summary

While Large Language Models (LLMs) have demonstrated mastery in knowledge retrieval and basic coding, their capacity for "higher-order cognitive thinking"—the complex mental processes required for graduate-level research and scientific discovery—remains a subject of intense debate. This study benchmarks OpenAI’s o1-preview model against human experts across seven cognitive domains. The results indicate that o1-preview significantly outperforms humans in five of these areas, including systematic and scientific reasoning, while exhibiting a notable deficit in adaptive, unstructured problem-solving.

The Challenge of Cognitive Benchmarking

The primary problem addressed by this research is the gap between standardized test performance and genuine cognitive ability. Most AI evaluations rely on datasets like MMLU, which often test rote knowledge. To determine if AI can transition from a "knowledge assistant" to a "cognitive peer," the researchers utilized validated psychological and educational instruments originally designed for human assessment.

Methods

The researchers conducted a comparative performance analysis using the following framework:

  • Iterative Prompting: The model was tested using zero-shot and role-based prompts (e.g., "respond as a postgraduate student") to align with the educational levels of human cohorts.
  • Human Benchmarking: AI scores were compared against data from peer-reviewed studies involving undergraduates, postgraduates, and subject-matter experts.
  • Statistical Validation: Z-scores and one-sample t-tests were employed to determine the statistical significance of performance gaps.
  • Scoring Rubrics: Open-ended tasks, such as critical thinking essays, were evaluated using the same rigorous rubrics applied to human participants.

Datasets and Human Baselines

The study utilized "gold standard" instruments for human cognitive assessment, including the Ennis-Weir Critical Thinking Essay Test (EWCTET), the Lake Urmia Vignette (LUV) for systematic thinking, and the Test of Scientific Literacy Skills (TOSLS).

Human benchmark data for Critical Thinking assessments used in the study.

As shown in the table above, the human benchmarks for critical thinking were derived from diverse cohorts, providing a robust baseline for comparing the AI's performance on the EWCTET.

Key Results

The comparative analysis revealed that o1-preview excels in domains with clear underlying logic or rules. The model achieved a near-perfect score on the TOSLS (0.99), surpassing biology experts (0.91), and more than doubled the human mean in systematic thinking.

Comparison of o1-preview and human mean performance across seven cognitive domains.

Performance Highlights:

  • Critical Thinking: o1-preview scored 24.33 on the EWCTET, significantly higher than the postgraduate mean of 18.39.
  • Systematic Thinking: The model achieved a score of 46.1, compared to the human mean of 20.08.
  • Logical Reasoning: It achieved 90% accuracy on LogiQA, outperforming the human average of 86%.
  • Scientific Reasoning: It reached 0.99 on the TOSLS, exceeding the highest human expert scores.

Key Insights and the "Problem-Solving Paradox"

One of the most striking findings was the model's mastery of "Feedback Loops" in systematic thinking, achieving a Z-score of 6.53. This suggests the model is far superior to humans at mapping interconnected causal relationships. Furthermore, in creative thinking tasks, the AI’s originality score (2.98) significantly beat the human mean (1.74), challenging the notion that AI is incapable of novelty.

However, the study identified a critical failure point: the "Problem-Solving Paradox." Despite high scores in algorithmic thinking, the model failed significantly in the unstructured problem-solving sub-dimension of Computational Thinking (Z-score of -4.25). This indicates that while the AI can follow complex "recipes" perfectly, it struggles when the environment is messy or instructions are ill-defined.

Limitations and Future Outlook

The study notes several caveats, including the conversion of visual graphs to text descriptions for the text-only o1-preview model and the relatively small sample size of AI trials. Future research must focus on "holistic assessments" that incorporate emotional intelligence, ethical reasoning, and multimodal inputs to fully map the boundaries of AI cognition. While o1-preview has reached human-level performance in structured reasoning, its limitations in adaptive problem-solving suggest it is not yet a complete replacement for human higher-order thought.

Summary updated

Abstract

<title>Abstract</title> This study evaluates the performance of OpenAI’s o1-preview model in higher-order cognitive domains, including critical thinking, systematic thinking, computational thinking, data literacy, creative thinking, logical reasoning, and scientific reasoning. Using established benchmarks, we compared the o1-preview models’s performance to human participants from diverse educational levels. o1-preview achieved a mean score of 24.33 on the Ennis-Weir Critical Thinking Essay Test (EWCTET), surpassing undergraduate (13.8) and postgraduate (18.39) participants (z = 1.60 and 0.90, respectively). In systematic thinking, it scored 46.1 ± 4.12 on the Lake Urmia Vignette, significantly outperforming the human mean (20.08 ± 8.13, z = 3.20). For data literacy, o1-preview scored 8.60 ± 0.70 on Merk et al.’s “Use Data” dimension, compared to the human post-test mean of 4.17 ± 2.02 (z = 2.19). On creative thinking tasks, the model achieved originality scores of 2.98 ± 0.73, higher than the human mean of 1.74 (z = 0.71). In logical reasoning (LogiQA), it outperformed humans with 90% ± 10 accuracy versus 86% ± 6.5 (z = 0.62). For scientific reasoning, it achieved near-perfect performance (0.99 ± 0.12) on the TOSLS" exceeding the highest human scores of 0.85 ± 0.13 (z = 1.78). While o1-preview excelled in structured tasks, it showed limitations in problem-solving and adaptive reasoning. These results demonstrate the potential of AI to complement education in structured assessments but highlight the need for ethical oversight and refinement for broader applications.
BibTeX
@misc{latif2024can,
  author = {Ehsan Latif and Yifan Zhou and Shuchen Guo and Yizhu Gao and Lehong Shi and Matthew Nyaaba and Arne Bewerdorff and Xiantong Yang and Xiaoming Zhaı},
  title = {{Can OpenAI o1 outperform humans in higher-order cognitive thinking?}},
  year = {2024},
  eprint = {2412.05753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.05753},
}
Figure