nbgrader: A Tool for Creating and Grading Assignments in the Jupyter Notebook

Project Jupyter, Douglas Blank, David Bourgin, Alexander Brown, Matthias Bussonnier, Jonathan Frederic, Brian Granger, Thomas L. Griffiths, Jessica B. Hamrick, Kyle Kelley, Michael Pacer, Logan Page, Fernando Pérez, Benjamin Ragan-Kelley, Jordan W. Suchow, Carol WillingJournal of Open Source Education 2019 · DOI

Abstract

nbgrader is a flexible tool for creating and grading assignments in the Jupyter Notebook. It allows instructors to create a single, master copy of an assignment, including tests and canonical solutions. From the master copy, a student version is generated without the solutions, thus obviating the need to maintain two separate versions. nbgrader also automatically grades submitted assignments by executing the notebooks and storing the results of the tests in a database. After auto-grading, instructors can manually grade free responses and provide partial credit using the formgrader Jupyter Notebook extension. Finally, instructors can use nbgrader to leave personalized feedback for each student's submission, including comments as well as detailed error information.

Introduction

As computational methods become integral to diverse fields such as bioinformatics, social sciences, and engineering, the Jupyter Notebook has emerged as a premier tool for interactive instruction. However, scaling the use of notebooks for graded assignments introduces significant logistical challenges. Instructors often face three primary "friction points": the difficulty of maintaining synchronized student and solution versions of assignments, the impossibility of manually grading code at scale, and the administrative burden of distributing and collecting files. nbgrader was developed to address these issues, transforming Jupyter Notebooks into a comprehensive educational ecosystem.

System Architecture and Methods

nbgrader is designed as a modular system that integrates directly into the Jupyter environment. Its workflow is centered around several core components:

  • The Master Copy and Metadata: Instructors maintain a single "source of truth" notebook. By utilizing cell metadata, nbgrader distinguishes between instructional content, solution cells, and test cells.
  • Assignment Generation: A command-line tool processes the master copy to generate a student-facing version. This process automatically strips out canonical solutions and hidden tests, ensuring consistency across versions.
  • Automated Grading: Upon submission, the autograder executes the student notebooks. It validates the code against hidden test cells and records the results in a local database, significantly reducing the manual workload for objective coding tasks.
  • The Formgrader Interface: Recognizing that qualitative "free-response" questions require human judgment, the formgrader extension provides a web interface for instructors. This allows for rapid manual grading, the awarding of partial credit, and the provision of personalized feedback.
  • JupyterHub Integration: For large-scale deployments, nbgrader integrates with JupyterHub, enabling a "one-click" workflow where students can fetch and submit assignments directly within their browser-based environment.

Adoption and Impact

While not a traditional experimental study, the efficacy of nbgrader is demonstrated through its widespread adoption. Since its inception in 2014, the tool has been implemented at major global institutions, including UC Berkeley, Cal Poly, and the University of Edinburgh. As of May 2018, over 10,000 nbgrader-based notebooks were identified on GitHub, covering a vast array of academic disciplines from mechanical engineering to bioinformatics. This adoption highlights the tool's versatility and its ability to meet the needs of diverse educational contexts.

Key Insights: Literate Grading

The core philosophy of nbgrader is "literate grading." It acknowledges that while code execution can be automated, the interpretation of results—the "why" behind the code—remains a critical pedagogical element that requires human intervention. By hiding grading logic in metadata, the tool maintains a clean interface for students while providing a robust framework for evaluators. This integration reduces "tooling fatigue," allowing students to focus on subject matter rather than the mechanics of file management or external submission portals.

Results and Extensions

The implementation of nbgrader has successfully enabled Jupyter-based assignments in courses with hundreds of students, a feat previously unattainable without prohibitive human effort. Beyond its core grading functionality, the tool has evolved to support:

  • Multi-course management on shared JupyterHub instances.
  • Configurable grading policies, including customizable late penalties and the ability to control the visibility of test results.
  • Interoperability with Learning Management Systems (LMS) like Canvas or Blackboard through file import/export capabilities.

Conclusion and Future Work

nbgrader significantly lowers the barrier to using interactive notebooks in large-scale education. Future development aims to further streamline the instructor-student communication loop and enhance the overall learning experience by minimizing administrative overhead. Ongoing efforts also focus on expanding LMS integration to ensure nbgrader can be seamlessly incorporated into existing institutional IT infrastructures.

Summary updated

Abstract

nbgrader is a flexible tool for creating and grading assignments in the Jupyter Notebook nbgrader allows instructors to create a single, master copy of an assignment, including tests and canonical solutions. From the master copy, a student version is generated without the solutions, thus obviating the need to maintain two separate versions. nbgrader also automatically grades submitted assignments by executing the notebooks and storing the results of the tests in a database. After auto-grading, instructors can manually grade free responses and provide partial credit using the formgrader Jupyter Notebook extension. Finally, instructors can use nbgrader to leave personalized feedback for each student's submission, including comments as well as detailed error information.
BibTeX
@article{jupyter2019nbgrader,
  author = {Project Jupyter and Douglas Blank and David Bourgin and Alexander Brown and Matthias Bussonnier and Jonathan Frederic and Brian Granger and Thomas L. Griffiths and Jessica B. Hamrick and Kyle Kelley and Michael Pacer and Logan Page and Fernando Pérez and Benjamin Ragan-Kelley and Jordan W. Suchow and Carol Willing},
  title = {{nbgrader: A Tool for Creating and Grading Assignments in the Jupyter Notebook}},
  journal = {Journal of Open Source Education},
  year = {2019},
  doi = {10.21105/jose.00032},
}
Figure