Sketching is a fundamental tool for spatial reasoning in STEM disciplines, yet it is often underutilized in large-scale classrooms due to the significant logistical challenges of grading hand-drawn work. This paper, presented at AAAI-18, introduces "Sketch Worksheets," an AI-powered system designed to provide students with immediate feedback while drastically reducing the grading burden on instructors.
The Challenge of Sketching in STEM
In fields like Geoscience, sketching is more than an artistic exercise; it is a method for conceptualizing complex spatial problems, such as groundwater flow or the folding of rock layers. However, instructors often face a "grading bottleneck." Manually evaluating hundreds of sketches is time-consuming, and the delay in returning feedback often means students miss the critical window for correcting misconceptions.
Traditional AI for sketching typically focuses on simple object recognition (e.g., identifying a square). In STEM, however, the spatial relationships and intent behind the shapes are paramount. A circle might represent a cell, a planet, or a pipe cross-section depending on the context. The authors address this by creating an "open-domain" system that understands the underlying structure of a sketch.
System Architecture: CogSketch and SME
The Sketch Worksheets system is built upon CogSketch, which bypasses the recognition problem by requiring students to label their drawings, known as glyphs. By labeling a circle as "Earth's Core," the student provides the semantic context, allowing the AI to focus on spatial and structural analysis.
The core of the system is the Structure-Mapping Engine (SME). SME utilizes a cognitive model of analogy to compare a student's sketch against an instructor's "gold standard" solution. It evaluates three primary components:
- Entity Glyphs: The objects drawn.
- Relation Glyphs: The interactions between objects (e.g., "Layer A is above Layer B").
- Quantitative Ink Constraints: Spatial accuracy, such as ensuring a traced fault line stays within a specific tolerance.
When a student requests feedback, the SME identifies structural mismatches and provides hyperlinked advice, enabling real-time iteration and learning.
Real-World Deployments and Results
The system was tested in two distinct academic environments:
- Geoscience: 26 worksheets were deployed in introductory geology courses, collecting data from nearly 500 students.
- Knowledge Representation (KR): Used in an upper-level Computer Science course for logic diagrams and knowledge graphs.
Key Findings
- Efficiency Gains: The system achieved a massive speedup in grading. In the Geoscience deployment, grading 196 paper worksheets took an instructor 5 hours. In contrast, grading 66 digital Sketch Worksheets took only 7.5 minutes—a more than 10x improvement in efficiency.
- Versatility: The system successfully transitioned from highly spatial tasks (Geoscience) to abstract logical encoding (Computer Science), proving its broad applicability.
- Student Engagement: Students relied heavily on the automated tutor, averaging 40 feedback requests for complex assignments, which served as a vital on-the-spot resource.
Limitations and Future Work
Despite its success, the system faced hardware hurdles; small tablet screens sometimes made it difficult for students to manage complex sketches and read feedback simultaneously. Additionally, while grading is nearly instantaneous, authoring high-quality worksheets remains a significant time investment for instructors, initially taking up to 1.5 weeks per assignment.
Future research will focus on refining the SME to handle highly complex logic diagrams where ambiguity can lead to incorrect feedback. Furthermore, the authors emphasize the need for randomized controlled trials to definitively measure the long-term learning gains of Sketch Worksheets compared to traditional paper-based methods.
Summary updated