Abstract: In this work, we focus on generating graphical representations of noisy, instructional videos for video understanding. We propose a self-supervised, interpretable approach that does not require any annotations for graphical representations, which would be expensive and time consuming to collect. We attempt to overcome "black box" learning limitations by presenting Semantic Video Graph or SVGraph, a multi-modal approach that utilizes narrations for semantic interpretability of the learned graphs. SVGraph 1) relies on the agreement between multiple modalities to learn a unified graphical structure with the help of cross-modal attention and 2) assigns semantic interpretation with the help of Semantic-Assignment, which captures the semantics from video narration. We perform experiments on multiple datasets and demonstrate the interpretability of SVGraph in semantic graph learning.
The Challenge of Instructional Video Understanding
Understanding instructional videos, such as "How to change a shower head," is inherently difficult because they are often noisy, lengthy, and vary significantly in quality and perspective. While scene graphs have successfully represented objects and relationships in static images, adapting them to video introduces three primary hurdles: the prohibitive cost of frame-by-frame annotation, the need to model temporal evolution, and the lack of interpretability in standard "black box" deep learning models.
The SVGraph Framework
SVGraph is a self-supervised framework designed to transform video understanding into a structured, human-readable format. It leverages the inherent redundancy in instructional videos—where the audio and visual streams often reinforce each other—to generate semantic graphs without manual labels. The pipeline consists of three main modules:
- Cross-Modal Attention: The system extracts visual features via a 3D CNN and audio features via a 2D CNN. A two-branch attention mechanism allows the visual stream to "attend" to the audio and vice versa, highlighting key moments where narration matches action (e.g., a speaker saying "use the pliers" while picking them up).
- Semantic Attention & Node Embedding: Using Automatic Speech Recognition (ASR), the model embeds narration words via Word2Vec. It then uses fused multi-modal features to select the most relevant words to serve as graph nodes.
- Message Passing & Semantic Assignment: Relationships (edges) are learned through a Message-Passing Neural Network (MPNN). Crucially, the Semantic Assignment mechanism tracks the contribution of original words to the final nodes, allowing the model to output a graph that humans can interpret.
The model is trained using a self-supervised triplet loss objective, minimizing the distance between a video and its augmented version while maximizing the distance from unrelated videos.
Experimental Results and Insights
The authors evaluated SVGraph on four diverse datasets: HowTo100M, COIN, YouCook2, and UCF101.
Key Findings: * Multi-modal Superiority: Integrating audio and video significantly outperformed single-modality approaches. On the YouCook2 dataset, the joint model achieved a Precision@5 of 9.1%, compared to 6.5% for video-only models. * Task Differentiation: Using the ROUGE-1 metric to measure node overlap, the model demonstrated high specificity. Graphs from the same task showed 82-86% overlap, while graphs from different tasks showed only 35-37% overlap, proving the model learns task-specific semantics rather than generic representations. * Human Interpretability: In a user study with 30 participants, users correctly matched generated graphs to their corresponding videos 79.6% of the time, confirming that the learned structures are intuitively meaningful.
Conclusion and Future Work
SVGraph demonstrates that cross-modal agreement can serve as a powerful supervisory signal for learning structured representations. While the model currently relies on ASR quality and does not yet explicitly label edge types (e.g., "acts-upon"), it provides a scalable, interpretable alternative to traditional video understanding methods. Future work may involve more granular temporal modeling to capture sub-activities in complex, long-form procedures.
Summary updated