STIVi: Turning Perspective Sketching Videos into Interactive Tutorials

Capucine Nghiem, Adrien Bousseau, Msc. Mark Sypesteyn, Jan Willem Hoftijzer, Maneesh Agrawala, Theophanis Tsandilas

TL;DR

This paper introduces STIVi, an interactive system that transforms static perspective sketching tutorials into dynamic learning experiences. By automatically linking video frames with transcript commentary, STIVi allows users to navigate, explore variations, and better understand drawing concepts at their own pace, addressing key limitations of traditional video-based learning.

What Problem Does This Paper Address?

Learning to draw, especially complex subjects like perspective sketching, through traditional instructional videos presents several challenges. Videos play at a fixed pace, making it difficult for learners to pause, replay, and practice fundamental concepts without missing crucial information. Students often struggle to divide their attention between the video and their own drawing canvas, leading to missed strokes or explanations. Furthermore, standard videos typically show only a limited number of examples, hindering the learner's ability to generalize concepts to different scenarios. This paper tackles these issues by creating an interactive system that enhances instructional videos for perspective drawing.

Methods

STIVi employs a semi-automatic pipeline to convert existing instructional videos into interactive tutorials. The core of the approach involves:

  • Visual Extraction: Analyzing video frames to extract individual pen strokes and infer geometric properties like line orientation and convergence towards vanishing points.
  • Text Extraction: Transcribing the instructor's audio commentary and identifying keywords related to drawing elements and their properties using NLP techniques.
  • Correspondence Building: Linking visual elements (strokes, lines, planes) with their corresponding textual descriptions in the transcript. This is achieved by calculating a similarity score based on temporal alignment and geometric compatibility. Instructors can then manually refine these suggested correspondences.
  • Interactive Augmentation: Leveraging the structured visual-textual data to augment the video with:
  • Navigation Landmarks: Highlighting key concepts on the video timeline and within the transcript.
  • Visual Highlights: Synchronizing visual cues (e.g., colored outlines, shapes) with spoken commentary to draw attention to specific drawing elements.
  • Interactive Exploration: Enabling users to manipulate elements like vanishing points and horizon lines to explore variations of the drawing demonstration and understand how perspective rules affect the geometry.

Datasets

The paper mentions the creation of a dataset for perspective drawing exercises. Two industrial design teachers created four standard beginner-level exercises, resulting in eight instructional videos. Additionally, five online videos were processed for testing the extraction pipeline. The paper does not introduce a new benchmark dataset but rather uses existing instructional videos.

Key Insights and Ideas

  • Bridging Modalities is Key: The core insight is that by tightly coupling visual drawing actions with their textual explanations, a much richer and more interactive learning experience can be created than with either modality alone.
  • Structured Data Enables Interaction: Extracting and structuring information about drawing elements (lines, planes, vanishing points) and their relationships from video and transcript allows for novel interactions like direct navigation via keywords or visual elements.
  • Semi-Automation for Scalability: A semi-automatic pipeline significantly reduces the burden on instructors to create interactive content, making it more feasible to convert existing video resources.
  • Non-Linear Learning is Crucial for Practice: The evaluation highlights that learners benefit from the ability to jump between concepts, explore variations, and revisit specific parts of the tutorial, supporting diverse learning styles and practice needs.

Key Results

A user study with 12 participants compared STIVi against a baseline video player. Subjective evaluations indicated that STIVi significantly helped participants:

  • Navigate the video more effectively: Participants found it easier to locate specific concepts and jump between sections.
  • Follow instructions: The visual highlights and transcript links aided in keeping track of the instructor's actions.
  • Understand concepts: The interactive exploration features, particularly for perspective manipulation, were beneficial.

Interaction logs revealed that STIVi promoted non-linear navigation, with users exploring the tutorial in a different order than the original video. Participants appreciated the multiple ways information was presented, catering to different learning styles.

Limitations and Future Work

  • Focus on Block Shapes: The current system is primarily demonstrated for perspective drawing of block shapes. Extending it to more organic forms (like faces, as shown in a mock-up) would require new domain-specific visual and textual vocabularies and more advanced image processing.
  • On-Paper Drawing Challenges: Experimentation with on-paper drawing videos revealed additional complexities (e.g., occluding hands, shadows) that require more sophisticated image processing, suggesting this area needs further development.
  • Automation of Pipeline: While semi-automatic, the pipeline still requires manual corrections for stroke extraction, transcript refinement, and correspondence assignment. Future work could focus on further automating these steps, potentially through data-driven approaches and larger annotated datasets.
  • Drawing Assistance on Canvas: Participants suggested applying STIVi's augmentations directly to the drawing canvas for real-time feedback and guidance, which is a promising avenue for future research.
  • Generalization of Exploration: The study focused on reproducing sketches. Further investigation is needed to see how the interactive exploration features are utilized for tasks requiring generalization to novel viewpoints or shapes.

Summary updated

Abstract

For design and art enthusiasts who seek to enhance their skills through instructional videos, following drawing instructions while practicing can be challenging. STIVi presents perspective drawing demonstrations and commentary of prerecorded instructional videos as interactive drawing tutorials that students can navigate and explore at their own pace. Our approach involves a semi-automatic pipeline to assist instructors in creating STIVi content by extracting pen strokes from video frames and aligning them with the accompanying audio commentary. Thanks to this structured data, students can navigate through transcript and in-video drawing, refer to provided highlights in both modalities to guide their navigation, and explore variations of the drawing demonstration to understand fundamental principles. We evaluated STIVi's interactive tutorials against a regular video player. We observed that our interface supports non-linear learning styles by providing students alternative paths for following and understanding drawing instructions.
BibTeX
@inproceedings{nghiem2024stivi,
  author = {Capucine Nghiem and Adrien Bousseau and Msc. Mark Sypesteyn and Jan Willem Hoftijzer and Maneesh Agrawala and Theophanis Tsandilas},
  title = {{STIVi: Turning Perspective Sketching Videos into Interactive Tutorials}},
  booktitle = {Graphics Interface},
  year = {2024},
  doi = {10.1145/3670947.3670969},
}

Loading related content...

Figure