Abstract
Video Question Answering (VideoQA) requires a sophisticated understanding of both visual and textual modalities. Traditional methods often rely on word-level alignment, which can lead to ambiguity when individual words are mapped to irrelevant visual features. We present the Semantic Composition and Alignment Network (SCAN), a framework that leverages syntactic dependency trees to construct a hypergraph of word compositions. By representing complex textual concepts as hyperedges and employing Optimal Transport for cross-modal mapping, SCAN achieves a sparse and precise alignment between language and video, significantly improving performance on major benchmarks.
Introduction
A fundamental challenge in Video Question Answering is cross-modal alignment—accurately mapping natural language queries to specific objects or actions within a video sequence. Most existing approaches attempt to align individual words to video regions. However, words in isolation are frequently ambiguous. For instance, in the question "What is the girl in green sitting on?", a word-level model might incorrectly align the word "green" with a green object in the background rather than the girl's attire.
The true semantic unit is often a composition of words, such as "girl in green." To address this, we propose that models must identify and represent these multi-word compositions (textual concepts) and align them directly to the video, rather than relying on fragmented word-level signals.
Methodology: The SCAN Framework
The Semantic Composition and Alignment Network (SCAN) introduces a structured approach to modeling textual concepts through three primary stages:
1. Syntactic Hypergraph Construction
Rather than treating a question as a simple sequence of tokens, SCAN utilizes a syntactic dependency tree (generated via Stanza). Every subtree within this structure is considered a "meaningful composition." These compositions are modeled as hyperedges in a hypergraph, where individual words serve as nodes. This structure effectively captures non-adjacent but semantically related words, such as "girl... sitting."
2. Hypergraph Convolutional Networks (HCN)
To learn representations for these hyperedges, SCAN employs a Hypergraph Convolutional Network. This component aggregates information from constituent word nodes into a single vector that represents a complex textual concept.
3. Alignment via Optimal Transport (OT)
Standard dot-product attention mechanisms often result in dense, noisy alignments. SCAN instead utilizes Optimal Transport (OT) to map textual concepts to video frames. OT finds the most efficient mapping between the distribution of textual concepts and visual frames, resulting in a sparser and more precise alignment.
4. Cross-Modality-Aware Updating
Finally, the model iteratively updates the representations of both modalities by injecting information across the alignment. This ensures that the final features used for prediction are deeply informed by the relevant context from the opposing modality.
Experiments and Results
The SCAN model was evaluated on three prominent VideoQA benchmarks: TGIF-QA, MSVD-QA, and MSRVTT-QA.
- TGIF-QA: SCAN demonstrated superior performance on the Action and Transition sub-tasks, outperforming the HAIR baseline by 2.6% and 2.4%, respectively.
- MSVD-QA & MSRVTT-QA: The model achieved significant gains over the DualVGR model, with accuracy improvements of 3.3% and 4.5% on these respective datasets.
Qualitative analysis confirms that SCAN's phrase-level alignment allows it to ignore irrelevant background noise. For example, while a word-level model might focus on any frame containing a "net," SCAN correctly identifies the specific frames where a "ball gets into the net."
Conclusion
By shifting the focus from individual words to syntactically-derived textual concepts, SCAN provides a more robust mechanism for cross-modal reasoning in VideoQA. The use of hypergraphs to model semantic composition and Optimal Transport for sparse alignment addresses key limitations in existing attention-based models. Future work may involve reducing the dependency on external parsers and optimizing the computational efficiency of the Optimal Transport mechanism for longer video sequences.
Summary updated