Instructor-designed AI improves learning in a randomized trial across 19 universities

Jun Li, Robert Bray, Sébastien Martin, Nils Van den Steen, Andrew Wu, Xiao Ni, Arielle Anderer, Ekaterina Astashkina, Volodymyr Babich, Jean-Paul Baldwin, Hamsa Bastani, Christian Blanco, Oguzhan Celebi, Christopher Chen, Shi Chen, Benjamin Collier, Yao Cui, Pnina Feldman, Blair Flicker, Brian Han, Yichun Hu, Daniela Hurtado-Lange, Jane Jiang, Zhaohui Jiang, Ismail Kirci, Murat Köksalan, Zhen Lian, Hongyan Liang, Mengshi Lu, Lesley Meng, Ken Moon, Mi Kyong Newsom, Minje Park, Chris Parker, Anyan Qi, Meng Qi, Debjit Roy, Neha Sharma, Alp Sungu, Tom Tan, Necati Tereyagoglu, Hanu Tyagi, Sriram Venkataraman, Guihua Wang, Yixin Wang, Owen Wu, Zonghao Yang, Qiuping Yu, Sophie Yu, Jingwei Zhang, Wenchang Zhang, Ruihao ZhuarXiv 2026 · arXiv

TL;DR

A massive randomized controlled trial across 19 universities reveals that when instructors—not administrators—design AI tools for their specific classrooms, student learning outcomes improve by 0.32 standard deviations. This "frontline-driven" approach succeeds where generic, off-the-shelf chatbots often fail, proving that AI is most effective as a pedagogical scalpel rather than a one-size-fits-all hammer.

What Problem Does This Paper Address?

The rapid integration of Generative AI in higher education has sparked a "tectonic disruption" alarm. Early evidence suggested that generic chatbots (like base ChatGPT) might actually undermine learning by encouraging "metacognitive laziness"—where students offload critical thinking to the AI, leading to worse performance on exams.

Most existing studies on AI in education have been limited by small sample sizes, short durations, or a "top-down" implementation of generic tools. The authors of this paper sought to determine if a "bottom-up" approach—empowering instructors to design custom AI solutions tailored to their specific course content and student needs—could reverse this negative trend and actually enhance learning.

Methods

The researchers conducted a pre-registered randomized controlled trial (RCT) involving 7,200 students across 19 universities over three semesters (Spring 2025 to Spring 2026).

Key methodological features included:

  • Frontline-Driven Design: Instructors used a platform called "Faculty Twin" (co-developed with Google) to build custom AI agents. These weren't just "chatbots"; they were specialized tools for Socratic coaching, interactive simulations, and course navigation.
  • Difference-in-Differences (DiD) Design: The study compared student performance in "AI-treated" classes against historical benchmarks from the same instructors and courses, as well as against a control group that delayed AI implementation.
  • AI-Proof Assessments: To ensure the results measured actual learning rather than the AI's ability to take a test, outcomes were measured via proctored, in-person, pen-and-paper examinations where AI assistance was impossible.
  • Standardization: Learning scores were standardized to account for heterogeneity across different STEM subjects and institutions.

Datasets

Unlike a traditional ML paper that trains on a static dataset, the "data" here is the experimental output of a large-scale human trial:

  • Primary Sample: 55 applied STEM classes (statistics, machine learning, optimization, etc.).
  • Student Data: De-identified records for 7,200 students, including exam scores, LMS activity logs (for engagement), and end-of-term evaluations (for satisfaction).
  • Qualitative Data: 2,019 student survey responses and interviews with 25 participating instructors.

Key Insights and Ideas

  • The Power of the Socratic Method: One of the most successful AI archetypes was the "Socratic Coach." Instead of providing answers, these agents were prompted to ask guiding questions, forcing students to explain their reasoning—a core tenet of active learning.
  • Trust Through Tailoring: Students reported significantly higher trust in AI tools that were grounded specifically in their course's syllabus and materials. They valued "content tailored to their courses" (23.2% of positive responses) over generic AI knowledge.
  • Frontline Knowledge is Non-Transferable: The study argues that the "heterogeneous nature of the classroom" requires instructor discretion. A statistics professor in Michigan and a supply chain professor in Hong Kong face different pedagogical "pain points" that a centrally-mandated AI tool cannot address.
  • Neutrality on Engagement: Surprisingly, while learning improved, student engagement (attendance, LMS logins) and satisfaction scores remained largely unchanged. This suggests that AI improves the quality of study time rather than just the quantity of it.

Key Results

  • Learning Gains: The instructor-designed AI innovations significantly improved student learning. The student-level treatment effect was 0.32 standard deviations (SD), and the class-level estimate was 0.26 SD.
  • Contextualizing the Gain: For comparison, a 0.32 SD improvement is substantial; typical pre-K-12 randomized interventions yield a median improvement of only 0.10 SD.
  • Student Sentiment: Students were 16.1 times more likely to report that AI improved their educational experience than worsened it.
  • Efficiency: 14.5% of students highlighted "time savings," noting they could get "minuscule questions" answered by the AI instead of waiting for office hours, allowing them to focus on deeper concepts.

Limitations and Future Work

  • Disciplinary Bias: The study focused almost exclusively on applied STEM subjects. Whether these results translate to the humanities or social sciences—where "correct" answers are often more subjective—remains to be seen.
  • Persistence: Most classes were only observed for one or two terms. The researchers note that they cannot yet conclude if these learning gains persist over a student's entire four-year degree.
  • Granularity: While the study proves that instructor-led AI works, it lacked the statistical power to determine which specific types of AI innovations (e.g., simulations vs. coaching) were the most effective.
  • Future Direction: The authors suggest future work should focus on identifying the specific design features (prompts, guardrails, UI) that contribute most to these learning gains.

Summary updated

Abstract

Universities are alarmed by recent evidence that artificial intelligence (AI) can undermine learning, with students prescribed generic chatbots or centrally designed AI tools performing worse. In a pre-registered randomized controlled trial across 7,200 students at 19 universities over three semesters, we tested whether AI designed by frontline instructors can improve learning. We encouraged instructors to design AI solutions they believed would best help their students. Drawing on their unique knowledge of their classes and students, the instructors created a wide array of innovations. We show that these instructor-designed AI innovations improved student learning, measured by AI-proof assessments (proctored, in-person examinations). We found no change in student engagement, measured by course activity records, or satisfaction, measured by course evaluations. Our findings demonstrate that AI can improve student learning when its adoption is frontline-driven, with instructors empowered to design and refine AI solutions for their own classrooms.
BibTeX
@misc{li2026instructor,
  author = {Jun Li and Robert Bray and Sébastien Martin and Nils Van den Steen and Andrew Wu and Xiao Ni and Arielle Anderer and Ekaterina Astashkina and Volodymyr Babich and Jean-Paul Baldwin and Hamsa Bastani and Christian Blanco and Oguzhan Celebi and Christopher Chen and Shi Chen and Benjamin Collier and Yao Cui and Pnina Feldman and Blair Flicker and Brian Han and Yichun Hu and Daniela Hurtado-Lange and Jane Jiang and Zhaohui Jiang and Ismail Kirci and Murat Köksalan and Zhen Lian and Hongyan Liang and Mengshi Lu and Lesley Meng and Ken Moon and Mi Kyong Newsom and Minje Park and Chris Parker and Anyan Qi and Meng Qi and Debjit Roy and Neha Sharma and Alp Sungu and Tom Tan and Necati Tereyagoglu and Hanu Tyagi and Sriram Venkataraman and Guihua Wang and Yixin Wang and Owen Wu and Zonghao Yang and Qiuping Yu and Sophie Yu and Jingwei Zhang and Wenchang Zhang and Ruihao Zhu},
  title = {{Instructor-designed AI improves learning in a randomized trial across 19 universities}},
  year = {2026},
  eprint = {2611.19099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19099},
}
Figure