A Better Way to Attend: Attention with Trees for Video Question Answering

Hongyang Xue, Wenqing Chu, Zhou Zhao, Deng Cai

Abstract

We propose a new attention model for video question answering. The main idea of the attention models is to locate on the most informative parts of the visual data. The attention mechanisms are quite popular these days. However, most existing visual attention mechanisms regard the question as a whole. They ignore the word-level semantics where each word can have different attentions and some words need no attention. Neither do they consider the semantic structure of the sentences. Although the Extended Soft Attention (E-SA) model for video question answering leverages the word-level attention, it performs poorly on long question sentences. In this paper, we propose the heterogeneous tree-structured memory network (HTreeMN) for video question answering. Our proposed approach is based upon the syntax parse trees of the question sentences. The HTreeMN treats the words differently where the visual words are processed with an attention module and the verbal ones not. It also utilizes the semantic structure of the sentences by combining the neighbors based on the recursive structure of the parse trees. The understandings of the words and the videos are propagated and merged from leaves to the root. Furthermore, we build a hierarchical attention mechanism to distill the attended features. We evaluate our approach on two datasets. The experimental results show the superiority of our HTreeMN model over the other attention models especially on complex questions.

Introduction

Video Question Answering (Video-QA) is a challenging task where a model must "watch" a video and answer a natural language question about it. Most existing models treat the question as a single, flat string of text, encoding it into a single vector.

Traditional Video-QA models often treat the question as a flat sequence, leading to issues with long sentences and irrelevant visual attention.

This approach suffers from two major flaws. First, there is a loss of structure: linear models like standard LSTMs struggle with long, complex sentences, often losing the syntactic relationships between words. Second, it falls into the "Everything is Visual" Fallacy: standard attention mechanisms try to find visual counterparts for every word in a sentence. However, words like "is," "the," or "and" (verbal words) do not have a visual representation in the video. Forcing the model to find "attention" for these words introduces unnecessary noise into the reasoning process.

Methods: The HTreeMN Model

The proposed Heterogeneous Tree-Structured Memory Network (HTreeMN) moves away from linear processing in favor of a tree structure based on the question's actual grammar. The model follows a multi-step process to encode the question and attend to the video:

  1. Syntax Parse Trees: The model uses the StanfordParser to transform a question into a tree. Leaves represent individual words, while internal nodes represent phrases (e.g., Noun Phrases or Verb Phrases).
  2. Heterogeneous Nodes: The model classifies leaf nodes into two types. Visual Words (nouns, most verbs, and adjectives) trigger a temporal attention module to scan video frames. Verbal Words (linking verbs and conjunctions) are processed for semantic meaning but do not attempt to "look" at the video.
  3. Hierarchical Attention: Attention is applied recursively. As the model moves up the tree from leaves to the root, intermediate nodes (phrases) also use attention to distill features from the video, allowing for multi-step reasoning.
  4. Bottom-Up Encoding: Information flows from the words at the bottom, through the phrases, to the root of the tree, where the final answer is generated.
The HTreeMN model uses a syntax parse tree to guide visual attention, distinguishing between visual words (e.g., 'woman', 'cooking') and verbal words (e.g., 'is', 'the').

Experimental Evaluation

The model was evaluated on two large-scale Video-QA benchmarks: YouTube-QA, derived from annotated YouTube clips, and TGIF-QA, based on animated GIFs. These datasets focus on objects, locations, numbers, and temporal actions.

Key Results

The HTreeMN model consistently outperformed state-of-the-art baselines like the Extended Soft Attention (E-SA) model:

  • Overall Accuracy: On YouTube-QA, HTreeMN achieved an accuracy of 32.52%, compared to E-SA’s 27.03%.
  • Complex Questions: The most dramatic gains were seen in long questions (over 20 words). While linear models often failed on these, the tree-structured model maintained high accuracy by leveraging syntactic structure.
  • WUPS Score: Using the WUPS metric (which accounts for word similarity), HTreeMN showed a significant lead, suggesting its answers are more semantically correct even when not matching the ground truth exactly.

Discussion and Conclusion

By using grammar as a guide, HTreeMN naturally combines words into phrases, mimicking human language understanding and making it robust for complex questions. The paper demonstrates that calculating attention for words like "by" or "the" actually hurts performance; by skipping visual attention for these "verbal" words, the model reduces noise. Furthermore, the recursive nature of the tree allows for sophisticated multi-step reasoning.

While HTreeMN is superior in structure, it remains limited by visual recognition errors (e.g., misidentifying a "mouse" as a "rabbit"). Future work will focus on integrating more powerful visual backbones and applying tree-structured attention to other multimodal tasks like video captioning or visual dialog.

Summary updated

Abstract

We propose a new attention model for video question answering. The main idea of the attention models is to locate on the most informative parts of the visual data. The attention mechanisms are quite popular these days. However, most existing visual attention mechanisms regard the question as a whole. They ignore the word-level semantics where each word can have different attentions and some words need no attention. Neither do they consider the semantic structure of the sentences. Although the Extended Soft Attention (E-SA) model for video question answering leverages the word-level attention, it performs poorly on long question sentences. In this paper, we propose the heterogeneous tree-structured memory network (HTreeMN) for video question answering. Our proposed approach is based upon the syntax parse trees of the question sentences. The HTreeMN treats the words differently where the \textit{visual} words are processed with an attention module and the \textit{verbal} ones not. It also utilizes the semantic structure of the sentences by combining the neighbors based on the recursive structure of the parse trees. The understandings of the words and the videos are propagated and merged from leaves to the root. Furthermore, we build a hierarchical attention mechanism to distill the attended features. We evaluate our approach on two datasets. The experimental results show the superiority of our HTreeMN model over the other attention models especially on complex questions. Our code is available on github. Our code is available at https://github.com/ZJULearning/TreeAttention
BibTeX
@article{xue2018better,
  author = {Hongyang Xue and Wenqing Chu and Zhou Zhao and Deng Cai},
  title = {{A Better Way to Attend: Attention with Trees for Video Question Answering}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=xq2l-MF-9l},
}
Figure