TL;DR
This paper introduces "Sketch Worksheets," an educational software system that leverages sketch understanding to provide on-the-spot feedback for spatial learning domains, traditionally underserved by intelligent tutoring systems. By interpreting student sketches and comparing them to instructor solutions using analogy, the system aims to make learning spatial concepts more engaging and assessment more efficient.
What Problem Does This Paper Address?
Intelligent tutoring systems (ITS) have shown great promise in enhancing education, but their application to highly spatial disciplines like geology, engineering, and mathematics has been limited. A major bottleneck is the difficulty in understanding student-generated sketches. Unlike symbolic representations, sketches rely on rich conceptual structures and visual processing, making them challenging for computers to interpret in a human-like way. Traditional paper-based worksheets offer delayed feedback and difficult assessment, hindering effective learning in these domains. This paper addresses the need for an ITS that can effectively interpret and provide feedback on sketches in spatial learning contexts.
Methods
The core of the Sketch Worksheets system is CogSketch, an open-domain sketch understanding system. CogSketch moves beyond simple ink recognition by focusing on interpreting the conceptual meaning behind sketches. It uses a large knowledge base (derived from OpenCyc) to model rich conceptual structures and employs structure-mapping processes (specifically, the Structure-Mapping Engine, SME) for visual reasoning and analogy.
Key methodological aspects include:
- Glyph-based representation: Sketches are broken down into "glyphs" representing entities, relations, and annotations, which are then conceptually labeled.
- Authoring Environment: A specialized interface allows non-developers to create worksheets by defining problem statements, drawing instructor solution sketches, selecting relevant concepts for students, and providing custom coaching advice.
- Analogy-based Coaching: When a student requests feedback, their sketch is compared to the instructor's solution sketch using SME. The system identifies correspondences and differences to generate targeted, textual feedback, often highlighting areas of the sketch.
- Iterative Refinement: Students can repeatedly refine their sketches based on the feedback, fostering an iterative learning process.
- Automated Assessment Support: The system logs student interactions and provides tools for instructors to grade worksheets based on predefined rubrics, including quantitative comparisons and the presence of key entities and facts.
Datasets
Not discussed in the paper. The system is designed to be domain-general, with worksheets created by instructors for specific educational tasks.
Key Insights and Ideas
- Sketching as a Catalyst, Not a Requirement: The system acknowledges that in human-to-human sketching, recognition is a starting point, not the end goal. The focus is on understanding the meaning conveyed by the sketch, not just identifying pre-defined symbols.
- Analogy is Key for Spatial Reasoning Feedback: Using analogy (via SME) to compare student and instructor sketches provides a powerful mechanism for identifying and explaining differences in spatial understanding.
- Bridging the Gap for Spatial Domains: Sketch Worksheets offer a viable path to bring the benefits of ITS to disciplines that have been historically underserved due to the complexity of spatial data.
- Empowering Non-Developers: The authoring environment is crucial for widespread adoption, enabling domain experts (teachers) to create educational content without needing AI expertise.
- The Nuance of "Correctness": The pilot study highlighted that "correctness" in sketches can be nuanced. A sketch that doesn't perfectly match the instructor's ink might still convey the correct conceptual understanding, posing an interesting challenge for automated assessment.
Key Results
A pilot study in a college geoscience course in Fall 2009 demonstrated the feasibility and potential of Sketch Worksheets.
- Usability: Students were able to use the system and complete worksheet exercises.
- Learning Curve: Student performance improved over time, with a significant negative correlation between the time taken to complete a worksheet and the number of worksheets previously completed. Students also had fewer remaining feedback items on later worksheets, indicating increased self-sufficiency.
- Engagement: Students found the exercises engaging and desired more such assignments.
- Qualitative Findings: The system provided valuable insights into student understanding through logged data. However, some usability issues were identified, such as students entering numerical values directly into annotation labels instead of using the dedicated dialog.
Limitations and Future Work
- Usability Issues: Students sometimes struggled with inputting numerical data correctly, suggesting a need for clearer instructions and feedback.
- Annotation Mapping: Incorrect mapping of annotations in some cases led to issues with automated scoring, prompting a change in representation conventions.
- Nuanced Sketch Interpretation: Accurately assessing sketches that convey correct conceptual understanding despite minor deviations from the instructor's ink remains a challenge. Finding robust ways to express and detect such variations is an area for future work.
- Future Directions:
- More Sophisticated Coaching: Equipping the system with domain-specific models to provide even richer tutoring.
- Expanded Classroom Studies: Conducting more studies with a larger pool of instructors and students across various topics (physical sciences, biology).
- Lowering Authoring Barriers: Developing a natural language layer for concepts and facts to further simplify worksheet authoring.
- Broader Assessment Applications: Exploring the use of sketch comparisons for diagnosing student mental models.
Summary updated