Video Question Answering on Screencast Tutorials

Wentian Zhao, Seokhwan Kim, Ning Xu, Hailin JinIJCAI 2020 · DOI

Introduction

While Video Question Answering (VideoQA) has made significant strides in understanding natural scenes—such as identifying ingredients in a cooking video—applying these models to software tutorials presents a unique set of challenges. In a screencast, the "world" is a digital interface where actions are often subtle, text-heavy, and deeply rooted in software-specific logic. Researchers from Adobe and Amazon have introduced a new VideoQA task and dataset designed to bridge the gap between raw pixels and the underlying domain knowledge of complex applications like Photoshop.

The Challenge of Software Interfaces

Unlike "in-the-wild" videos, software tutorials require more than just visual observation. Users often ask questions that demand domain knowledge not explicitly shown on screen or mentioned in the audio. For instance, a user might see a dialog box and ask about a keyboard shortcut or the specific menu path required to reach that state.

Standard computer vision techniques, which excel at identifying physical objects, often struggle to differentiate between visually similar UI components. A "Save" dialog and a "Save As" dialog may look nearly identical to a standard CNN, yet they represent distinct software states with different functional implications.

Figure 5: A sequence showing the 'Save As' workflow, requiring specific menu navigation and dialog interaction.

Methodology: Fusing Vision, Text, and Knowledge

To address these challenges, the authors proposed a multi-modal architecture that integrates four information streams:

  1. Question & Transcript Encoding: Standard encoders process the user's natural language query and the narrated audio transcript.
  2. Visual Cue Extraction: The model moves beyond global frames by detecting specific UI components. While tool icons are identified using a ResNet classifier, text-heavy panels and dialogs are processed via a novel one-shot matching algorithm. This algorithm uses OCR to extract text, which is then embedded and matched against a library of known software states.
  3. Knowledge Grounding: This is the core innovation. Every potential answer is treated as an entity within a domain knowledge base (KB). Using DeepWalk graph embeddings, the model learns the relationships between software features (e.g., the hierarchical relationship between the "File" menu and "Save As").
  4. Dual Attention: The model employs a mechanism that coordinates "when" to look (temporal attention on the transcript) and "where" to look (spatial attention on the UI elements).
Figure 6: A sequence showing the 'Save' workflow, which visually mirrors the initial steps of 'Save As' but results in a different software action.

Key Results and Insights

The research demonstrates that context and knowledge are essential for navigating software complexity:

  • The Power of Knowledge: Incorporating Knowledge Graph embeddings (DeepWalk) provided the single largest performance boost, raising Recall@1 (R@1) from 45.8% (multi-modal without KB) to 55.9%.
  • Textual UI Recognition: Treating the UI as a collection of OCR-derived words proved far more effective than relying solely on visual features, as it allowed the model to distinguish between functionally different but visually similar dialogs.
  • Attention Synergy: The Dual Attention mechanism provided a 1.5% accuracy improvement, confirming that aligning temporal cues from the narration with spatial cues from the interface is vital for instructional tasks.

Conclusion

The introduction of this specialized VideoQA task highlights the limitations of general-purpose vision models when applied to structured, digital environments. By grounding visual cues in a domain-specific knowledge base, the researchers have created a framework that better understands the logic of software. Future work will focus on improving the robustness of automated UI detection to move these models closer to real-time, interactive help systems.

Summary updated

BibTeX
@inproceedings{zhao2020video,
  author = {Wentian Zhao and Seokhwan Kim and Ning Xu and Hailin Jin},
  title = {{Video Question Answering on Screencast Tutorials}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/148},
}
Figure