TL;DR
Despite faculty efforts to design "AI-proof" college assessments, state-of-the-art Large Language Models (LLMs) like GPT-5.4 still achieve average scores of 83–85% across diverse disciplines. While adding strategic complexity can hinder AI performance, many common "proofing" tactics—such as demanding specific output formats—fail to significantly impact model accuracy, signaling a profound challenge for traditional academic integrity.
What Problem Does This Paper Address?
The rapid adoption of LLMs has created an "academic-integrity challenge unlike any in history." While previous research evaluated LLMs on standardized tests (like the Bar exam or USMLE) or single-discipline datasets, this paper addresses the gap in how AI handles authentic, multidisciplinary college coursework specifically designed by faculty to be "AI-resistant."
The authors investigate whether "AI-proofing" strategies actually work, which features of a question constrain LLM performance, and how the availability of supplementary course materials (lecture slides, notes) changes the equation.
Methods
The researchers employed a multi-phase, crowdsourced methodology:
- Data Collection: They hosted a competition for faculty across 11 universities to submit "LLM-proof" questions. They collected 75 questions across 17 disciplines, including 11 pairs of "original" vs. "tweaked" (AI-resistant) variants.
- Answer Generation: They tested three frontier models: ChatGPT-5.4, Gemini 2.5 Flash, and Claude Sonnet 4.6. Answers were generated in two conditions: "zero-shot" (question only) and with supplementary course materials.
- Evaluation: They used an LLM-as-a-judge framework, where GPT-5.4 scored the 4,500 generated answers based on faculty-provided rubrics. This automated grading was validated against a sample of 102 human-graded responses to ensure reliability.
- Thematic Analysis: The authors analyzed faculty explanations to identify five core strategies used to resist AI.
Datasets
The study introduces a new, heterogeneous dataset of 75 college-level assessment questions:
- 61 "LLM-proof" questions: Intentionally designed by 37 faculty members to be difficult for AI.
- 14 Online-sourced questions: A control group of standard questions where AI resistance was unknown.
- 17 Disciplines: Ranging from Computer Science (44%) and Media Studies (13.3%) to Theology and Engineering.
- Diverse Formats: Includes reasoning/calculations, programming, written case studies, and graphical representations.
Key Insights and Ideas
- The "AI-Proofing" Paradox: Most strategies intended to make questions harder for AI didn't work. In fact, Strategy 5 (Demanding Specific Output Formats)—like requiring an IRAC structure or specific diagrams—resulted in the highest median scores (91-92%), likely because the structure provided the LLM with a clearer roadmap for the answer.
- Complexity is the Best Defense: Only Strategy 4 (Adding Strategic Complexity), which involves layered problem structures and deliberate misdirection, showed a consistent, though modest, reduction in LLM performance.
- Context Doesn't Always Help: Surprisingly, providing supplementary course materials only marginally improved LLM scores (a ~2% bump for GPT-5.4). This suggests that frontier models already possess enough general knowledge to handle specific course contexts.
- Model Variance: There are noteworthy performance gaps between models. GPT-5.4 consistently outperformed Gemini and Claude, particularly on the most difficult "AI-proof" questions.
Key Results
- High Baseline Performance: Even on questions designed to be "AI-proof," GPT-5.4 averaged 83.2% without extra materials and 85.3% with them.
- Online vs. Proofed: All models performed better on standard online-sourced questions than on faculty-tweaked ones, but the difference was small (roughly 4–6 points).
- Format Impact: LLMs struggled most with Reasoning steps / calculation (median ~83%) and Graphical representation (median ~86% but with high variability), while Programming tasks were handled with high proficiency (median 85–88%).
Limitations and Future Work
The authors acknowledge several limitations:
- Sample Size: 75 questions is a modest dataset, and the distribution across the 17 disciplines is uneven (heavily weighted toward Computer Science).
- Prompt Engineering: The study used a standard prompt for all models; it's possible that sophisticated prompt engineering by students could further increase these scores.
- Rapid Evolution: As LLM capabilities advance (e.g., better multimodal reasoning for graphical questions), current "AI-proofing" strategies will likely become even less effective.
Future Work: The authors suggest moving away from "prohibition-based" policies toward governance that integrates LLMs into the educational pipeline, such as requiring students to critically analyze AI-generated responses rather than just producing answers.
Summary updated