Abstract
Generative AI systems have rapidly advanced, with multimodal input capabilities enabling reasoning beyond text-based tasks. In education, these advancements could influence assessment design and question answering, presenting both opportunities and challenges. To investigate these effects, we introduce a high-quality dataset of 201 university-level STEM questions, manually annotated with features such as image type, role, problem complexity, and question format. Our study analyzes how these features affect generative AI performance compared to students. We evaluate four model families with five prompting strategies, comparing results to the average of 546 student responses per question. Although the best model correctly answers on average 58.5% of the questions using majority vote aggregation, human participants consistently outperform AI on questions involving visual components. Interestingly, human performance remains stable across question features but varies by subject, whereas AI performance is susceptible to both subject matter and question features. Finally, we provide actionable insights for educators, demonstrating how question design can enhance academic integrity by leveraging features that challenge current AI systems without increasing the cognitive burden for students.
Introduction
As Multimodal Large Language Models (MLLMs) become ubiquitous, educators face a dual challenge: AI can be a powerful personalized tutor, but it also threatens the integrity of traditional assessments. Most existing benchmarks evaluate AI models in a vacuum, making it hard to determine if a "hard" problem is difficult because of the subject matter or because of specific AI limitations. Researchers at EPFL sought to bridge this gap by directly comparing AI performance against historical student data, identifying specific "features" of STEM questions that make them "AI-resistant" without necessarily increasing the cognitive load for human students.
Methodology
The study employed a rigorous comparative framework using a manually curated dataset of 201 university-level STEM questions sourced from Bachelor’s and Master’s programs across 11 fields, including Quantum Physics, Microfabrication, and Electromagnetism.
- Human Baseline: Aggregated performance data from 546 student responses per question.
- AI Evaluation: Five state-of-the-art model families (including GPT-4o, Claude 3.7 Sonnet, and DeepSeek r1) were tested using five distinct prompting strategies.
- Prompting Innovation: Beyond standard zero-shot and Chain-of-Thought (CoT), the study tested "Image-First" prompts and a "Two-Stage" approach where one model describes the image for a second, reasoning-heavy model.
- Annotation: Questions were manually labeled for image type (diagram, plot, etc.), image purpose (crucial vs. supplemental), and problem complexity.
Key Results
The best-performing model, GPT-4o, achieved an average accuracy of 58.5%, trailing the human student average of 62.7%. However, performance varied drastically by subject matter. AI excelled in structured fields like Astronomy (78%) and Computer Science (78%), while struggling significantly in Quantum Physics (35%) and Chemistry (47%), where content-rich, complex images are prevalent.
The "Crucial Image" Barrier and Visual Reasoning
A primary finding of the study is the "Crucial Image" barrier. AI performance drops significantly when an image is "crucial" (containing information not present in the text). In contrast, human students maintain relatively stable performance regardless of whether the information is presented textually or visually. Humans excel at "physics-based reasoning" and real-world conventions—such as understanding force distributions in a diagram—whereas AI struggles with these implicit relationships.
Furthermore, AI performance tanks when a problem requires integrating more than two distinct concepts. While students handle this "interdependency" well, AI systems reach a complexity ceiling. Interestingly, as question length increases, student accuracy tends to decline due to cognitive load, whereas AI performance remains stable or improves, processing extended contexts effortlessly.
Discussion and Conclusion
The study highlights a significant gap in AI's ability to handle multi-concept integration and crucial visual data. While GPT-4o could identify the correct answer in up to 75.5% of cases with its best prompt, its lower "majority vote" accuracy (58.5%) suggests a lack of consistent retrieval.
For educators, these findings suggest that "AI-resilient" exams should focus on multi-concept integration and crucial visual data. Such designs challenge current AI systems without increasing the cognitive burden for students, though care must be taken to ensure accessibility for students with visual impairments who may rely on text descriptions similar to those used in AI prompting.
Summary updated