Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models

Sarah Zhang, Samuel Florin, Ariel N. Lee, Eamon Niknafs, Andrei Marginean, Annie Wang, Keith Tyser, Zad Chin, Yann Hicke, Nikhil Singh, Madeleine Udell, Yoon Kim, Tonio Buonassisi, Armando Solar-Lezama, Iddo Drori

TL;DR

This paper introduces a comprehensive dataset of MIT's Mathematics and EECS curriculum questions and solutions to evaluate Large Language Models (LLMs). They demonstrate that advanced LLMs like GPT-4, with sophisticated prompting techniques, can achieve near-perfect performance on this challenging curriculum, opening doors for LLMs in educational assessment and curriculum design.

What Problem Does This Paper Address?

Evaluating the true academic capabilities of LLMs beyond single exams has been a significant challenge due to the lack of comprehensive, curriculum-level datasets. Existing benchmarks often focus on specific subjects or standardized tests, failing to capture the breadth and depth required for a full undergraduate degree. This paper addresses this gap by creating a dataset covering all MIT Mathematics and EECS courses, enabling a systematic evaluation of LLMs' ability to master an entire academic program. This is crucial for understanding LLMs' potential in education, from student learning to curriculum development.

Methods

The authors employ a multi-pronged approach:

  • Dataset Curation: They meticulously collected and curated 4,550 questions and solutions from problem sets, midterms, and final exams across 30 MIT Mathematics and EECS courses. This involved manual correction for accuracy and quality.
  • LLM Benchmarking: They evaluated several state-of-the-art LLMs (GPT-4, GPT-3.5, StableVicuna, LLaMA) using various prompting techniques, including zero-shot, few-shot, chain-of-thought, and a novel "expert prompting" method.
  • Expert Prompting: This new technique involves asking the LLM to identify subject matter experts for a given question, then generating answers as if those experts wrote them, and finally combining these into a collective decision.
  • Automatic Grading: GPT-4 was used to automatically grade LLM responses, enabling a detailed performance breakdown.
  • Curriculum Analysis: Questions were embedded into a low-dimensional space to analyze relationships between courses and identify prerequisite structures.
  • Fine-tuning: An open-source LLM (LLaMA) was fine-tuned on their dataset to create a publicly available model (MIT-LLM).

Datasets

The primary dataset introduced in this paper is MITCOURSES, a collection of 4,550 questions and solutions from MIT's Mathematics and Electrical Engineering and Computer Science (EECS) undergraduate curriculum. This dataset is unique in its scope, covering an entire degree program rather than isolated courses or exams. They also mention using the ReClor dataset for validation of some of their findings.

Key Insights and Ideas

  • LLMs Can Master Entire Curricula: The results strongly suggest that advanced LLMs, particularly GPT-4, are capable of understanding and solving a significant portion of a rigorous undergraduate curriculum, moving beyond single-course mastery.
  • Prompt Engineering is Key: The effectiveness of LLMs is highly dependent on the prompting strategy. Sophisticated techniques like "expert prompting" and "chain-of-thought" significantly boost performance.
  • Curriculum Structure Revealed by Embeddings: Embedding questions allows for the discovery of implicit relationships and prerequisite structures between courses, offering a data-driven approach to curriculum design.
  • Meta-Questions for Deeper Understanding: The authors propose "meta-questions" where students evaluate LLM-generated answers, fostering critical thinking and a deeper understanding of the material rather than just rote memorization.

Key Results

  • GPT-3.5 solved approximately one-third of the MIT curriculum questions.
  • GPT-4, with advanced prompt engineering (including expert prompting and self-critique), achieved a perfect solve rate on a test set excluding image-based questions.
  • The "expert prompting" technique showed an improvement in performance compared to prior prompt engineering methods.
  • Fine-tuning an open-source LLM (LLaMA) on the MITCOURSES dataset improved its performance on reasoning tasks.
  • The analysis of question embeddings revealed relationships between courses, suggesting potential prerequisite structures.

Limitations and Future Work

  • Inference and Grading Speed: The paper notes that inference and automatic grading using GPT-4 are relatively slow, taking about a minute per question. This is attributed to the limited context window of 8k tokens in the models used at the time of writing.
  • Dataset Availability: To preserve its value as a benchmark, the MITCOURSES dataset is not publicly released but is available to researchers upon request via a data use agreement.
  • Future Work: The authors suggest that their approach can be extended to other universities and curricula. They also highlight the potential for LLMs to assist in curriculum design, identify knowledge gaps, and guide students in course selection. The development of new meta-questions is also a key area for future exploration.

Summary updated

Abstract

We curate a comprehensive dataset of 4,550 questions and solutions from problem sets, midterm exams, and final exams across all MIT Mathematics and Electrical Engineering and Computer Science (EECS) courses required for obtaining a degree. We evaluate the ability of large language models to fulfill the graduation requirements for any MIT major in Mathematics and EECS. Our results demonstrate that GPT-3.5 successfully solves a third of the entire MIT curriculum, while GPT-4, with prompt engineering, achieves a perfect solve rate on a test set excluding questions based on images. We fine-tune an open-source large language model on this dataset. We employ GPT-4 to automatically grade model responses, providing a detailed performance breakdown by course, question, and answer type. By embedding questions in a low-dimensional space, we explore the relationships between questions, topics, and classes and discover which questions and classes are required for solving other questions and classes through few-shot learning. Our analysis offers valuable insights into course prerequisites and curriculum design, highlighting language models' potential for learning and improving Mathematics and EECS education.
BibTeX
@misc{zhang2023exploring,
  author = {Sarah Zhang and Samuel Florin and Ariel N. Lee and Eamon Niknafs and Andrei Marginean and Annie Wang and Keith Tyser and Zad Chin and Yann Hicke and Nikhil Singh and Madeleine Udell and Yoon Kim and Tonio Buonassisi and Armando Solar-Lezama and Iddo Drori},
  title = {{Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models}},
  year = {2023},
  eprint = {2306.08997},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.08997},
}
Figure