Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison

Aymeric de Chillaz, Anna Sotnikova, Patrick Jermann, Antoine Bosselut

Abstract: As Generative AI (GenAI) becomes increasingly prevalent in educational settings, understanding its limitations in specialized academic domains is critical. This paper presents a comparative analysis between state-of-the-art multimodal models and university students on STEM assessments from EPFL. The study reveals that while AI models are nearing human parity in aggregate scores, they exhibit significant deficiencies in "crucial" visual reasoning and the integration of multiple interdependent concepts—areas where human intuition remains superior.

Introduction

The rapid advancement of Large Language Models (LLMs) has demonstrated their capability to pass standardized tests like the Bar exam. However, these benchmarks often evaluate AI in isolation. This research addresses whether AI failures in STEM are due to model limitations or inherent problem difficulty by conducting a head-to-head comparison with university students. The focus is on multimodal assessments where visual elements like diagrams and plots are essential for problem-solving.

Methodology

The researchers evaluated models including GPT-4o, o1-mini, Qwen 2.5 72B VL, DeepSeek r1, and Claude 3.7 Sonnet. Five prompting strategies were employed: Direct Zero-Shot, Chain-of-Thought (CoT), Image-First analysis, Two-Stage (textual description followed by reasoning), and Simultaneous/Recursive loops. Performance was benchmarked against a manually curated dataset of 201 university-level questions across 11 STEM fields, paired with historical data from an average of 546 students per question.

Analysis of AI Performance

The study identified several key "blind spots" for AI. A primary finding is the "Crucial Image Hurdle": AI performance declines sharply when an image is indispensable to the solution, whereas human performance remains relatively stable or even improves with the presence of visual context.

Figure 5: Comparison of model and student accuracy based on the necessity of visual information. AI models struggle significantly when images are "crucial" to the solution.

Furthermore, AI performance varies drastically by subject. Models excel in fields with well-defined rules or high textual data availability, such as Computer Science and Astronomy, but falter in subjects requiring deep physical intuition or complex spatial reasoning, like Quantum Physics and Electromagnetism.

Figure 4: Heatmap showing accuracy across different STEM subjects for both students and the best-performing AI model.

Key Results

The top-performing model, GPT-4o, achieved a majority-vote accuracy of 58.5%, trailing the human average of 62.7%. Interestingly, while consistent accuracy was lower, models provided the correct answer in at least one prompting trial for 75.5% of questions, indicating a "retrieval gap" where the model possesses the necessary knowledge but fails to apply it consistently across different reasoning paths.

Discussion and Conclusion

The findings suggest that AI is currently more robust against "wall-of-text" problems than humans, who suffer from cognitive fatigue. However, AI lacks the intuition-driven reasoning required for interpreting directional trends or implicit physical constraints. To ensure academic integrity and effective assessment, educators are encouraged to design "AI-resistant" exams that emphasize multi-concept integration and rely on crucial visual data. Future research will explore human-like reasoning strategies, such as elimination, to bridge the current performance gap.

Summary updated

Abstract

Generative AI systems have rapidly advanced, with multimodal input capabilities enabling reasoning beyond text-based tasks. In education, these advancements could influence assessment design and question answering, presenting both opportunities and challenges. To investigate these effects, we introduce a high-quality dataset of 201 university-level STEM questions, manually annotated with features such as image type, role, problem complexity, and question format. Our study analyzes how these features affect generative AI performance compared to students. We evaluate four model families with five prompting strategies, comparing results to the average of 546 student responses per question. Although the best model correctly answers on average 58.5 % of the questions using majority vote aggregation, human participants consistently outperform AI on questions involving visual components. Interestingly, human performance remains stable across question features but varies by subject, whereas AI performance is susceptible to both subject matter and question features. Finally, we provide actionable insights for educators, demonstrating how question design can enhance academic integrity by leveraging features that challenge current AI systems without increasing the cognitive burden for students.
BibTeX
@misc{chillaz2025challenges,
  author = {Aymeric de Chillaz and Anna Sotnikova and Patrick Jermann and Antoine Bosselut},
  title = {{Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison}},
  year = {2025},
  eprint = {2507.03013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.03013},
}
Figure