Modeling Semantic Composition with Syntactic Hypergraph for Video Question Answering

Zenan Xu, Wanjun Zhong, Qinliang Su, Zijing Ou, Fuwei Zhang

Abstract

Video Question Answering (VideoQA) requires a sophisticated understanding of both visual and textual modalities. Traditional methods often rely on word-level alignment, which can lead to ambiguity when individual words are mapped to irrelevant visual features. We present the Semantic Composition and Alignment Network (SCAN), a framework that leverages syntactic dependency trees to construct a hypergraph of word compositions. By representing complex textual concepts as hyperedges and employing Optimal Transport for cross-modal mapping, SCAN achieves a sparse and precise alignment between language and video, significantly improving performance on major benchmarks.

Introduction

A fundamental challenge in Video Question Answering is cross-modal alignment—accurately mapping natural language queries to specific objects or actions within a video sequence. Most existing approaches attempt to align individual words to video regions. However, words in isolation are frequently ambiguous. For instance, in the question "What is the girl in green sitting on?", a word-level model might incorrectly align the word "green" with a green object in the background rather than the girl's attire.

Figure 1: Comparison between word-level and phrase-level alignment. Word-level alignment for "green" might focus on background elements, while phrase-level alignment for "girl in green" correctly identifies the subject.

The true semantic unit is often a composition of words, such as "girl in green." To address this, we propose that models must identify and represent these multi-word compositions (textual concepts) and align them directly to the video, rather than relying on fragmented word-level signals.

Methodology: The SCAN Framework

The Semantic Composition and Alignment Network (SCAN) introduces a structured approach to modeling textual concepts through three primary stages:

1. Syntactic Hypergraph Construction

Rather than treating a question as a simple sequence of tokens, SCAN utilizes a syntactic dependency tree (generated via Stanza). Every subtree within this structure is considered a "meaningful composition." These compositions are modeled as hyperedges in a hypergraph, where individual words serve as nodes. This structure effectively captures non-adjacent but semantically related words, such as "girl... sitting."

2. Hypergraph Convolutional Networks (HCN)

To learn representations for these hyperedges, SCAN employs a Hypergraph Convolutional Network. This component aggregates information from constituent word nodes into a single vector that represents a complex textual concept.

3. Alignment via Optimal Transport (OT)

Standard dot-product attention mechanisms often result in dense, noisy alignments. SCAN instead utilizes Optimal Transport (OT) to map textual concepts to video frames. OT finds the most efficient mapping between the distribution of textual concepts and visual frames, resulting in a sparser and more precise alignment.

Figure 5: The overall architecture of the SCAN framework, illustrating multi-modal feature extraction, cross-modality-aware representation learning, and the final prediction stage.

4. Cross-Modality-Aware Updating

Finally, the model iteratively updates the representations of both modalities by injecting information across the alignment. This ensures that the final features used for prediction are deeply informed by the relevant context from the opposing modality.

Experiments and Results

The SCAN model was evaluated on three prominent VideoQA benchmarks: TGIF-QA, MSVD-QA, and MSRVTT-QA.

  • TGIF-QA: SCAN demonstrated superior performance on the Action and Transition sub-tasks, outperforming the HAIR baseline by 2.6% and 2.4%, respectively.
  • MSVD-QA & MSRVTT-QA: The model achieved significant gains over the DualVGR model, with accuracy improvements of 3.3% and 4.5% on these respective datasets.

Qualitative analysis confirms that SCAN's phrase-level alignment allows it to ignore irrelevant background noise. For example, while a word-level model might focus on any frame containing a "net," SCAN correctly identifies the specific frames where a "ball gets into the net."

Conclusion

By shifting the focus from individual words to syntactically-derived textual concepts, SCAN provides a more robust mechanism for cross-modal reasoning in VideoQA. The use of hypergraphs to model semantic composition and Optimal Transport for sparse alignment addresses key limitations in existing attention-based models. Future work may involve reducing the dependency on external parsers and optimizing the computational efficiency of the Optimal Transport mechanism for longer video sequences.

Summary updated

Abstract

A key challenge in video question answering is how to realize the cross-modal semantic alignment between textual concepts and corresponding visual objects. Existing methods mostly seek to align the word representations with the video regions. However, word representations are often not able to convey a complete description of textual concepts, which are in general described by the compositions of certain words. To address this issue, we propose to first build a syntactic dependency tree for each question with an off-the-shelf tool and use it to guide the extraction of meaningful word compositions. Based on the extracted compositions, a hypergraph is further built by viewing the words as nodes and the compositions as hyperedges. Hypergraph convolutional networks (HCN) are then employed to learn the initial representations of word compositions. Afterwards, an optimal transport based method is proposed to perform cross-modal semantic alignment for the textual and visual semantic space. To reflect the cross-modal influences, the cross-modal information is incorporated into the initial representations, leading to a model named cross-modality-aware syntactic HCN. Experimental results on three benchmarks show that our method outperforms all strong baselines. Further analyses demonstrate the effectiveness of each component, and show that our model is good at modeling different levels of semantic compositions and filtering out irrelevant information.
BibTeX
@misc{xu2022modeling,
  author = {Zenan Xu and Wanjun Zhong and Qinliang Su and Zijing Ou and Fuwei Zhang},
  title = {{Modeling Semantic Composition with Syntactic Hypergraph for Video Question Answering}},
  year = {2022},
  eprint = {2205.06530},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.06530},
}
Figure