Abstract
Understanding web instructional videos is an essential branch of video understanding. Unlike unconstrained long videos, instructional videos are more structured, following step-by-step procedures. This paper studies reasoning on instructional videos via question-answering (QA) and introduces YouQuek, an annotated QA dataset based on YouCook2. The questions in YouQuek require logical reasoning in the temporal dimension rather than just single-frame cues. To address the lack of effective representations for modeling long videos, the authors propose a Recurrent Graph Convolutional Network (RGCN) that captures both temporal order and relational information. Experiments show that RGCN, especially when combined with human-annotated descriptions, outperforms existing models in QA accuracy.
Introduction: Beyond Short-Term Action Recognition
Most current video AI models excel at "short-term action recognition"—identifying a three-second clip of someone performing a simple task like "chopping an onion." However, instructional videos, such as cooking tutorials, are long, structured, and procedural. Understanding them requires higher-order inference: Why was this ingredient added now? How many steps occurred before the oven was turned on?
Existing benchmarks often lack the logical "connective tissue" needed to reason across a multi-minute video. There is a significant gap in testing a machine's ability to understand the logic of a sequence rather than just the pixels in isolated frames. This paper addresses this gap by focusing on reasoning-based VideoQA.
The YouQuek Dataset
To facilitate research in this area, the authors introduce YouQuek, built on top of the YouCook2 cooking dataset. It consists of 15,355 manually collected QA pairs across over 2,000 videos. Unlike previous datasets that often rely on auto-generated questions, YouQuek is human-annotated to ensure questions require genuine logical reasoning to solve.
Questions in the dataset are categorized into six reasoning styles:
- Counting: Quantifying actions or ingredients.
- Time & Order: Determining when or in what sequence events occurred.
- Taste & Property: Identifying changes in ingredient states or flavors.
- Complex: Multi-step reasoning across the entire video duration.
Methodology: Recurrent Graph Convolutional Networks (RGCN)
The primary architectural contribution is the RGCN, designed to balance sequential processing with relational reasoning. Standard RNNs track sequences but struggle with long-range dependencies, while GCNs model complex relationships but often ignore temporal order.
The RGCN uses a "swap" operation to pass information between an RNN pathway and a GCN pathway. This allows the model to maintain a sense of time while simultaneously reasoning about how different steps in a recipe relate to one another. Additionally, the model incorporates "Question-to-Segment" attention to focus on relevant video parts and explores multimodal fusion by integrating human-annotated descriptions and YouTube transcripts.
Key Insights and Results
- The Power of the Hybrid: The RGCN-SA (with attention) achieved a top accuracy of 40.3%, significantly outperforming naive RNNs (34.7%) and the state-of-the-art MAC model (34.8%). Graph-based components were particularly effective for "Counting" and "Taste" questions.
- The "Common Sense" Factor: A "Human Quiz" revealed that people could answer 52.8% of questions correctly using only common sense, without watching the video. This highlights how instructional reasoning is deeply tied to world knowledge (e.g., knowing to wash vegetables before cooking).
- Transcripts vs. Descriptions: While human-written descriptions significantly boosted performance, auto-generated transcripts actually hindered it. The authors attribute this to the "noise" in speech—chefs often discuss unrelated topics, which distracts the model from the visual procedure.
- The Human-AI Gap: Despite these advances, a massive gap remains. Humans score 97% when watching the video, compared to the AI's 40.3%.
Future Directions
While the RGCN is a step forward, it still lacks the "life experience" humans use to fill in the blanks. Future work will focus on incorporating external knowledge graphs to provide models with common sense. Additionally, improving the filtering of spoken narratives in transcripts and developing end-to-end models that can simultaneously segment and reason about videos are key priorities.
Summary updated