TL;DR
While Multimodal Large Language Models (MLLMs) can identify qualitative features in physics graphs, they are prone to "hallucinating" credit for severely distorted or nonsensical drawings. This study uses synthetic stress testing to demonstrate that while AI can distinguish complex physical states, it should not be trusted to grade student sketches without human oversight.
What Problem Does This Paper Address?
Grading open-ended student work, such as hand-drawn kinematics graphs, is a time-intensive task that requires nuanced reasoning. While MLLMs like GPT-4o show promise in automating this, their "reasoning" is a black box. The authors investigate a critical gap: Does an AI grader award points based on actual visual evidence, or is it merely "filling in the blanks" based on the rubric?
In physics education, distinguishing between an object at rest for a moment versus a sustained stop is vital. If an AI cannot reliably see these distinctions in a messy student sketch, it risks providing misleading feedback and unfair grades.
Methods
The researchers employed a "stress testing" framework using two primary methods to generate synthetic student responses for four kinematics exercises:
- Forward Generation (Distortion): Starting with a "perfect" reference graph, they used Stable Diffusion v1.5 to introduce visual noise, jitter, and fragmentation, mimicking the imperfections of human sketching.
- Reverse Generation (Morphing): They took a "donor" graph (a correct answer for a different problem) and progressively transformed it toward the target reference solution.
These images were then fed to MLLM-based grading assistants (specifically versions identified in the paper as GPT-5.1 and GPT-6 Astra) along with a rubric containing qualitative requirements translated into numerical cutoffs (e.g., $dx/dt > 0.1$).
Datasets
The study utilized a synthetic corpus of 6,000 graded images (1,500 per exercise). The exercises covered four scenarios:
- Constantly increasing position.
- Forward motion, followed by a stop, then reverse motion (Position vs. Time).
- Acceleration from rest followed by constant velocity.
- Forward motion, followed by a stop, then reverse motion (Velocity vs. Time).
Key Insights and Ideas
- The "Hallucination" of Credit: One of the most striking findings was that a severely distorted velocity-time graph—featuring disconnected fragments and unreadable axes—received a perfect 10/10 score. The model's explanation claimed it saw a "sustained flat segment on the $v=0$ line," even though the image was visually incoherent.
- Qualitative vs. Quantitative Tension: The models were surprisingly good at qualitative physics (e.g., distinguishing a smooth turning point from a sustained stop). However, they struggled to apply numerical rubric cutoffs to unnumbered axes, often inventing their own coordinate conventions on the fly.
- Total Scores are Deceptive: A high total score often masked failures in specific criteria. For instance, a model might award points for "forward motion" while completely missing that the "stopping interval" was physically absent in the drawing.
Key Results
- Successes: In cleaner "morphing" sequences, the models correctly withheld credit when a graph lacked positive curvature (acceleration) or failed to show a sustained stop.
- Failures: In the "Robust-to-Noise" group, models frequently awarded full credit to images that an instructor would likely reject as uninterpretable.
- Surrogate Limitations: The authors trained a ResNet-50 surrogate model to predict the MLLM's grades, but it achieved only a weak Pearson correlation of 0.28. This suggests that the MLLM’s grading logic is not easily captured by standard computer vision architectures.
Limitations and Future Work
The primary limitation is the use of synthetic data; while Stable Diffusion can mimic "jitter," it may not capture the specific types of conceptual errors real students make. Furthermore, the study did not test for score stability (whether the model gives the same grade to the same image twice).
The authors propose a "human-in-the-loop" workflow where the AI provides "stress signals" to instructors, highlighting ambiguous cases that require manual review. Future work should focus on defining clearer coordinate conventions for AI rubrics to prevent the model from making arbitrary numerical assumptions.
Summary updated