SVGraph: Learning Semantic Graphs from Instructional Videos

Madeline C. Schiappa, Yogesh S. Rawat

This paper presents SVGraph, a framework designed to transform the "black box" of video understanding into an interpretable, structured format. By leveraging self-supervised learning across multiple modalities, the authors generate semantic graphs from instructional videos without requiring a single manual annotation.

TL;DR

SVGraph is a self-supervised framework that automatically generates interpretable semantic graphs from instructional videos by fusing visual, audio, and textual (ASR) data. It uses cross-modal attention to identify key moments and a "Semantic Assignment" mechanism to map learned features back to human-readable words, providing a structured representation of complex activities without the need for manual labels.

What Problem Does This Paper Address?

Understanding instructional videos (e.g., "How to change a shower head") is challenging because they are often noisy, long, and vary significantly in quality and perspective. While "scene graphs" have been successful in the image domain for representing objects and their relationships, they are difficult to adapt to video.

The primary hurdles are:

  1. Annotation Cost: Obtaining frame-by-frame labels for entities and relationships in video is prohibitively expensive.
  2. Temporal Dimension: Unlike static images, videos require modeling how actions and objects evolve over time.
  3. Interpretability: Most deep learning models for video are "black boxes" that output a classification label without explaining why or how the steps of a task relate to one another.

Methods

SVGraph employs a multi-modal, self-supervised pipeline consisting of three main modules:

  1. Cross-Modal Attention: The system extracts visual features (via a 3D CNN) and audio features (via a 2D CNN). It uses a two-branch attention mechanism where the visual stream "attends" to the audio stream and vice versa. This highlights segments where the narration and visual cues reinforce each other (e.g., a person saying "use the pliers" while picking them up).
  2. Semantic Attention & Node Embedding: The model takes words from the video's narration (via ASR) and embeds them using Word2Vec. It then uses the fused multi-modal features to select the most relevant words to serve as nodes in a graph.
  3. Message Passing & Semantic Assignment: To learn relationships (edges), the model uses a Message-Passing Neural Network (MPNN) implemented via temporal and node-wise convolutions. Crucially, the Semantic Assignment mechanism tracks which original words contributed most to the final graph nodes, allowing the model to output a graph that humans can actually read.
  4. Self-Supervised Objective: The model is trained using a triplet loss. It learns to minimize the distance between a video and its augmented version while maximizing the distance from a different, randomly selected video.

Datasets

The authors evaluated SVGraph on four diverse datasets:

  • HowTo100M: A massive dataset of narrated instructional videos; a subset of ~20,000 videos was used for training.
  • COIN: Contains 11,827 instructional videos across 180 different tasks.
  • YouCook2: A large-scale cooking dataset used to test the model's ability to handle fine-grained, task-oriented procedures.
  • UCF101: A standard action recognition dataset used primarily for ablation studies and testing robustness.

Key Insights and Ideas

  • Redundancy is a Feature: In instructional videos, the audio and visual streams are often redundant (the speaker describes what they are doing). SVGraph turns this redundancy into a supervisory signal, using the "agreement" between modalities to find the most important parts of the video.
  • Local vs. Global Semantics: The model is surprisingly good at distinguishing between similar tasks. For example, when comparing two videos about making pizza, the graphs correctly identify unique ingredients (e.g., "olives" vs. "sugar") rather than just settling for a generic "pizza" representation.
  • Interpretability via Reverse Mapping: By maintaining a link between high-level embeddings and the original ASR word tokens, the model provides a "paper trail" for its decisions, making the resulting graph a literal map of the video's content.

Key Results

  • Multi-modal Superiority: Using both audio and video significantly outperformed single-modality versions. On the YouCook2 dataset, the joint model achieved a Precision@5 of 9.1%, compared to 6.5% for video-only.
  • Task Differentiation: The authors used the ROUGE-1 metric (typically used in NLP) to measure node overlap. Graphs from the same task showed high overlap (~82-86%), while graphs from different tasks showed low overlap (~35-37%), proving the model learns task-specific semantics.
  • User Study: In a study with 30 participants, users were able to correctly match a generated graph to its corresponding video approximately 79.6% of the time, demonstrating that the graphs are intuitively meaningful to humans.

Limitations and Future Work

  • ASR Dependency: The model relies heavily on the quality of Automatic Speech Recognition. If the narration is poor, missing, or misaligned, the semantic quality of the graph nodes suffers.
  • Edge Semantics: While the model identifies that a relationship exists between nodes (edges), it does not yet explicitly label the type of relationship (e.g., "is-part-of" or "acts-upon") in the same way it labels nodes.
  • Temporal Resolution: The authors suggest that future work could involve more granular temporal modeling to better capture the exact sequence of sub-activities in even longer, more complex videos.

Summary updated

Abstract

In this work, we focus on generating graphical representations of noisy, instructional videos for video understanding. We propose a self-supervised, interpretable approach that does not require any annotations for graphical representations, which would be expensive and time consuming to collect. We attempt to overcome "black box" learning limitations by presenting Semantic Video Graph or SVGraph, a multi-modal approach that utilizes narrations for semantic interpretability of the learned graphs. SVGraph 1) relies on the agreement between multiple modalities to learn a unified graphical structure with the help of cross-modal attention and 2) assigns semantic interpretation with the help of Semantic-Assignment, which captures the semantics from video narration. We perform experiments on multiple datasets and demonstrate the interpretability of SVGraph in semantic graph learning.
BibTeX
@inproceedings{schiappa2022svgraph,
  author = {Madeline C. Schiappa and Yogesh S. Rawat},
  title = {{SVGraph: Learning Semantic Graphs from Instructional Videos}},
  booktitle = {BigMM},
  year = {2022},
  doi = {10.1109/bigmm55396.2022.00014},
}

Loading related content...

Figure