Abstract: As Generative AI (GenAI) becomes increasingly prevalent in educational settings, understanding its limitations in specialized academic domains is critical. This paper presents a comparative analysis between state-of-the-art multimodal models and university students on STEM assessments from EPFL. The study reveals that while AI models are nearing human parity in aggregate scores, they exhibit significant deficiencies in "crucial" visual reasoning and the integration of multiple interdependent concepts—areas where human intuition remains superior.
Introduction
The rapid advancement of Large Language Models (LLMs) has demonstrated their capability to pass standardized tests like the Bar exam. However, these benchmarks often evaluate AI in isolation. This research addresses whether AI failures in STEM are due to model limitations or inherent problem difficulty by conducting a head-to-head comparison with university students. The focus is on multimodal assessments where visual elements like diagrams and plots are essential for problem-solving.
Methodology
The researchers evaluated models including GPT-4o, o1-mini, Qwen 2.5 72B VL, DeepSeek r1, and Claude 3.7 Sonnet. Five prompting strategies were employed: Direct Zero-Shot, Chain-of-Thought (CoT), Image-First analysis, Two-Stage (textual description followed by reasoning), and Simultaneous/Recursive loops. Performance was benchmarked against a manually curated dataset of 201 university-level questions across 11 STEM fields, paired with historical data from an average of 546 students per question.
Analysis of AI Performance
The study identified several key "blind spots" for AI. A primary finding is the "Crucial Image Hurdle": AI performance declines sharply when an image is indispensable to the solution, whereas human performance remains relatively stable or even improves with the presence of visual context.
Furthermore, AI performance varies drastically by subject. Models excel in fields with well-defined rules or high textual data availability, such as Computer Science and Astronomy, but falter in subjects requiring deep physical intuition or complex spatial reasoning, like Quantum Physics and Electromagnetism.
Key Results
The top-performing model, GPT-4o, achieved a majority-vote accuracy of 58.5%, trailing the human average of 62.7%. Interestingly, while consistent accuracy was lower, models provided the correct answer in at least one prompting trial for 75.5% of questions, indicating a "retrieval gap" where the model possesses the necessary knowledge but fails to apply it consistently across different reasoning paths.
Discussion and Conclusion
The findings suggest that AI is currently more robust against "wall-of-text" problems than humans, who suffer from cognitive fatigue. However, AI lacks the intuition-driven reasoning required for interpreting directional trends or implicit physical constraints. To ensure academic integrity and effective assessment, educators are encouraged to design "AI-resistant" exams that emphasize multi-concept integration and rely on crucial visual data. Future research will explore human-like reasoning strategies, such as elimination, to bridge the current performance gap.
Summary updated