Abstract
nbgrader is a flexible tool for creating and grading assignments in the Jupyter Notebook. It allows instructors to create a single, master copy of an assignment, including tests and canonical solutions. From the master copy, a student version is generated without the solutions, thus obviating the need to maintain two separate versions. nbgrader also automatically grades submitted assignments by executing the notebooks and storing the results of the tests in a database. After auto-grading, instructors can manually grade free responses and provide partial credit using the formgrader Jupyter Notebook extension. Finally, instructors can use nbgrader to leave personalized feedback for each student's submission, including comments as well as detailed error information.
Introduction
As computational methods become integral to diverse fields such as bioinformatics, social sciences, and engineering, the Jupyter Notebook has emerged as a premier tool for interactive instruction. However, scaling the use of notebooks for graded assignments introduces significant logistical challenges. Instructors often face three primary "friction points": the difficulty of maintaining synchronized student and solution versions of assignments, the impossibility of manually grading code at scale, and the administrative burden of distributing and collecting files. nbgrader was developed to address these issues, transforming Jupyter Notebooks into a comprehensive educational ecosystem.
System Architecture and Methods
nbgrader is designed as a modular system that integrates directly into the Jupyter environment. Its workflow is centered around several core components:
- The Master Copy and Metadata: Instructors maintain a single "source of truth" notebook. By utilizing cell metadata,
nbgraderdistinguishes between instructional content, solution cells, and test cells. - Assignment Generation: A command-line tool processes the master copy to generate a student-facing version. This process automatically strips out canonical solutions and hidden tests, ensuring consistency across versions.
- Automated Grading: Upon submission, the autograder executes the student notebooks. It validates the code against hidden test cells and records the results in a local database, significantly reducing the manual workload for objective coding tasks.
- The Formgrader Interface: Recognizing that qualitative "free-response" questions require human judgment, the
formgraderextension provides a web interface for instructors. This allows for rapid manual grading, the awarding of partial credit, and the provision of personalized feedback. - JupyterHub Integration: For large-scale deployments,
nbgraderintegrates with JupyterHub, enabling a "one-click" workflow where students can fetch and submit assignments directly within their browser-based environment.
Adoption and Impact
While not a traditional experimental study, the efficacy of nbgrader is demonstrated through its widespread adoption. Since its inception in 2014, the tool has been implemented at major global institutions, including UC Berkeley, Cal Poly, and the University of Edinburgh. As of May 2018, over 10,000 nbgrader-based notebooks were identified on GitHub, covering a vast array of academic disciplines from mechanical engineering to bioinformatics. This adoption highlights the tool's versatility and its ability to meet the needs of diverse educational contexts.
Key Insights: Literate Grading
The core philosophy of nbgrader is "literate grading." It acknowledges that while code execution can be automated, the interpretation of results—the "why" behind the code—remains a critical pedagogical element that requires human intervention. By hiding grading logic in metadata, the tool maintains a clean interface for students while providing a robust framework for evaluators. This integration reduces "tooling fatigue," allowing students to focus on subject matter rather than the mechanics of file management or external submission portals.
Results and Extensions
The implementation of nbgrader has successfully enabled Jupyter-based assignments in courses with hundreds of students, a feat previously unattainable without prohibitive human effort. Beyond its core grading functionality, the tool has evolved to support:
- Multi-course management on shared JupyterHub instances.
- Configurable grading policies, including customizable late penalties and the ability to control the visibility of test results.
- Interoperability with Learning Management Systems (LMS) like Canvas or Blackboard through file import/export capabilities.
Conclusion and Future Work
nbgrader significantly lowers the barrier to using interactive notebooks in large-scale education. Future development aims to further streamline the instructor-student communication loop and enhance the overall learning experience by minimizing administrative overhead. Ongoing efforts also focus on expanding LMS integration to ensure nbgrader can be seamlessly incorporated into existing institutional IT infrastructures.
Summary updated