Generative AI and the Crisis of Higher Education Assessments: Examining 'AI-Proofing' Strategies for College-Level Assessments

Bonam Mingole, Davis Yadav, Hui Min Lee, S. Shyam Sundar, Amulya YadavarXiv 2026 · arXiv

TL;DR

Despite faculty efforts to design "AI-proof" college assessments, state-of-the-art Large Language Models (LLMs) like GPT-5.4 still achieve average scores of 83–85% across diverse disciplines. While adding strategic complexity can hinder AI performance, many common "proofing" tactics—such as demanding specific output formats—fail to significantly impact model accuracy, signaling a profound challenge for traditional academic integrity.

What Problem Does This Paper Address?

The rapid adoption of LLMs has created an "academic-integrity challenge unlike any in history." While previous research evaluated LLMs on standardized tests (like the Bar exam or USMLE) or single-discipline datasets, this paper addresses the gap in how AI handles authentic, multidisciplinary college coursework specifically designed by faculty to be "AI-resistant."

The authors investigate whether "AI-proofing" strategies actually work, which features of a question constrain LLM performance, and how the availability of supplementary course materials (lecture slides, notes) changes the equation.

Methods

The researchers employed a multi-phase, crowdsourced methodology:

  1. Data Collection: They hosted a competition for faculty across 11 universities to submit "LLM-proof" questions. They collected 75 questions across 17 disciplines, including 11 pairs of "original" vs. "tweaked" (AI-resistant) variants.
  2. Answer Generation: They tested three frontier models: ChatGPT-5.4, Gemini 2.5 Flash, and Claude Sonnet 4.6. Answers were generated in two conditions: "zero-shot" (question only) and with supplementary course materials.
  3. Evaluation: They used an LLM-as-a-judge framework, where GPT-5.4 scored the 4,500 generated answers based on faculty-provided rubrics. This automated grading was validated against a sample of 102 human-graded responses to ensure reliability.
  4. Thematic Analysis: The authors analyzed faculty explanations to identify five core strategies used to resist AI.
Fig. 1. Overview of the methodology used to assess LLMs’ performance on multidisciplinary instructor-crafted assessment questions

Datasets

The study introduces a new, heterogeneous dataset of 75 college-level assessment questions:

  • 61 "LLM-proof" questions: Intentionally designed by 37 faculty members to be difficult for AI.
  • 14 Online-sourced questions: A control group of standard questions where AI resistance was unknown.
  • 17 Disciplines: Ranging from Computer Science (44%) and Media Studies (13.3%) to Theology and Engineering.
  • Diverse Formats: Includes reasoning/calculations, programming, written case studies, and graphical representations.

Key Insights and Ideas

  • The "AI-Proofing" Paradox: Most strategies intended to make questions harder for AI didn't work. In fact, Strategy 5 (Demanding Specific Output Formats)—like requiring an IRAC structure or specific diagrams—resulted in the highest median scores (91-92%), likely because the structure provided the LLM with a clearer roadmap for the answer.
  • Complexity is the Best Defense: Only Strategy 4 (Adding Strategic Complexity), which involves layered problem structures and deliberate misdirection, showed a consistent, though modest, reduction in LLM performance.
  • Context Doesn't Always Help: Surprisingly, providing supplementary course materials only marginally improved LLM scores (a ~2% bump for GPT-5.4). This suggests that frontier models already possess enough general knowledge to handle specific course contexts.
  • Model Variance: There are noteworthy performance gaps between models. GPT-5.4 consistently outperformed Gemini and Claude, particularly on the most difficult "AI-proof" questions.

Key Results

  • High Baseline Performance: Even on questions designed to be "AI-proof," GPT-5.4 averaged 83.2% without extra materials and 85.3% with them.
  • Online vs. Proofed: All models performed better on standard online-sourced questions than on faculty-tweaked ones, but the difference was small (roughly 4–6 points).
  • Format Impact: LLMs struggled most with Reasoning steps / calculation (median ~83%) and Graphical representation (median ~86% but with high variability), while Programming tasks were handled with high proficiency (median 85–88%).
Fig. 2. Distribution of LLM grades across three LLMs.

Limitations and Future Work

The authors acknowledge several limitations:

  • Sample Size: 75 questions is a modest dataset, and the distribution across the 17 disciplines is uneven (heavily weighted toward Computer Science).
  • Prompt Engineering: The study used a standard prompt for all models; it's possible that sophisticated prompt engineering by students could further increase these scores.
  • Rapid Evolution: As LLM capabilities advance (e.g., better multimodal reasoning for graphical questions), current "AI-proofing" strategies will likely become even less effective.

Future Work: The authors suggest moving away from "prohibition-based" policies toward governance that integrates LLMs into the educational pipeline, such as requiring students to critically analyze AI-generated responses rather than just producing answers.

Summary updated

Abstract

Higher education is confronting an academic-integrity challenge unlike any in its history. Students can now outsource homework, take-home assignments, and online examinations to large language models (LLMs) that complete them in seconds. Existing research on evaluating LLM performance on academic assessments has focused largely on standardized examinations, or on questions from college-level courses drawn from a single discipline. Far less is known about how LLMs perform on college-level assessment questions spanning multiple disciplines, that instructors have purposely designed to be AI-resistant, and about which features of such questions, if any, meaningfully constrain LLM performance. We address this gap by collecting 75 questions from authentic college coursework assessments, spanning 17 disciplines, which were contributed by 37 faculty members from ten U.S. universities and one U.K. university. Of these, 61 questions were intentionally designed by faculty members to be 'LLM-proof', while the remaining 14 questions were sourced from online courses. We then (i) evaluate three state-of-the-art LLMs' ability to correctly answer these questions with and without supplementary course materials; (ii) identify five strategies, through thematic analysis, that faculty used to make their questions harder for LLMs to solve; and (iii) test whether these strategies impact LLMs' ability to answer the questions accurately. Our experimental results show that the best-performing model scored approximately 83% without supplementary course materials and 85% with them, and we observed noteworthy differences across models, answer formats, and instructor tweaking strategies. We discuss implications for assessment design, academic-integrity policy, and the broader ethics of deploying LLMs in educational settings.
BibTeX
@misc{mingole2026generative,
  author = {Bonam Mingole and Davis Yadav and Hui Min Lee and S. Shyam Sundar and Amulya Yadav},
  title = {{Generative AI and the Crisis of Higher Education Assessments: Examining 'AI-Proofing' Strategies for College-Level Assessments}},
  year = {2026},
  eprint = {2611.21476},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21476},
}
Figure