A Better Way to Attend: Attention With Trees for Video Question Answering

Hongyang Xue, Wenqing Chu, Zhou Zhao, Deng Cai

TL;DR

Researchers have developed a "Heterogeneous Tree-Structured Memory Network" (HTreeMN) that significantly improves Video Question Answering (Video-QA). By using the syntax tree of a question to guide visual attention, the model distinguishes between "visual" words that need to look at the video and "verbal" words that provide structure, leading to much better performance on long, complex questions.

What Problem Does This Paper Address?

Video Question Answering (Video-QA) is a challenging task where a model must "watch" a video and answer a natural language question about it. Most existing models treat the question as a single, flat string of text, encoding it into a single vector.

The authors identify two major flaws in this approach:

  1. Loss of Structure: Linear models (like standard LSTMs) struggle with long, complex sentences, often losing the syntactic relationships between words.
  2. The "Everything is Visual" Fallacy: Standard attention mechanisms try to find visual counterparts for every word in a sentence. However, words like "is," "the," or "and" (verbal words) don't have a visual representation in the video. Forcing the model to find "attention" for these words introduces unnecessary noise.

Methods

The authors propose the HTreeMN model, which moves away from linear processing in favor of a tree structure based on the question's actual grammar.

  • Syntax Parse Trees: The model uses the StanfordParser to turn a question into a tree. The leaves are individual words, and the internal nodes represent phrases (like Noun Phrases or Verb Phrases).
  • Heterogeneous Nodes: This is the "secret sauce." The model classifies leaf nodes into two types:
  • Visual Words: Nouns, most verbs, and adjectives. These trigger a temporal attention module to scan the video frames.
  • Verbal Words: Linking verbs and conjunctions. These are processed for their semantic meaning but do not attempt to "look" at the video.
  • Hierarchical Attention: Attention isn't just for words. As the model moves up the tree from leaves to the root, intermediate nodes (phrases) also use attention to distill features from the video, allowing for multi-step reasoning.
  • Bottom-Up Encoding: Information flows from the words at the bottom, through the phrases, to the root of the tree, where the final answer is generated.

Datasets

The model was evaluated on two large-scale Video-QA benchmarks:

  • YouTube-QA: Derived from annotated YouTube video clips.
  • TGIF-QA: Based on the TGIF dataset, which contains animated GIFs and natural language descriptions. The questions in these datasets focus on objects, locations, numbers, and time (e.g., "What is the woman cooking?" or "How many people are dancing?").

Key Insights and Ideas

  • Grammar as a Guide: By following the syntax tree, the model naturally combines words into phrases and phrases into sentences, mimicking how humans understand language. This makes it much more robust for "long-tail" complex questions.
  • Noise Reduction: The paper demonstrates that calculating attention for words like "by" or "the" actually hurts performance. By labeling these as "verbal" and skipping the visual attention step for them, the model focuses its "eyes" only where it matters.
  • Recursive Reasoning: Because the tree is processed recursively, the model can perform a form of multi-step reasoning—narrowing down its focus on the video as it moves from simple words to complex phrases.

Key Results

The HTreeMN model consistently outperformed state-of-the-art baselines like the Extended Soft Attention (E-SA) model.

  • Overall Accuracy: On the YouTube-QA dataset, HTreeMN achieved an accuracy of 32.52%, compared to E-SA’s 27.03%.
  • Complex Questions: The most dramatic gains were seen in long questions (over 20 words). In these cases, linear models often failed completely, while the tree-structured model maintained high accuracy.
  • WUPS Score: Using the WUPS metric (which accounts for word similarity in answers), HTreeMN showed a significant lead, suggesting its answers are not just more accurate, but more semantically "correct" even when they don't match the ground truth exactly.

Limitations and Future Work

The authors noted that while their structural approach is superior, the model is still limited by recognition errors. For example, the model might correctly identify the action but misidentify the object (e.g., calling a "mouse" a "rabbit" or "lemon" an "orange").

Future work could involve:

  • Integrating more powerful visual backbones to reduce recognition errors.
  • Applying this tree-structured attention to other multimodal tasks like video captioning or visual dialog.
  • Exploring ways to automatically learn the "visual vs. verbal" distinction rather than relying on a fixed threshold of concreteness.

Summary updated

Abstract

We propose a new attention model for video question answering. The main idea of the attention models is to locate on the most informative parts of the visual data. The attention mechanisms are quite popular these days. However, most existing visual attention mechanisms regard the question as a whole. They ignore the word-level semantics where each word can have different attentions and some words need no attention. Neither do they consider the semantic structure of the sentences. Although the extended soft attention model for video question answering leverages the word-level attention, it performs poorly on long question sentences. In this paper, we propose the heterogeneous tree-structured memory network (HTreeMN) for video question answering. Our proposed approach is based upon the syntax parse trees of the question sentences. The HTreeMN treats the words differently where the visual words are processed with an attention module and the verbal ones not. It also utilizes the semantic structure of the sentences by combining the neighbors based on the recursive structure of the parse trees. The understandings of the words and the videos are propagated and merged from leaves to the root. Furthermore, we build a hierarchical attention mechanism to distill the attended features. We evaluate our approach on two data sets. The experimental results show the superiority of our HTreeMN model over the other attention models, especially on complex questions.
BibTeX
@article{xue2018better,
  author = {Hongyang Xue and Wenqing Chu and Zhou Zhao and Deng Cai},
  title = {{A Better Way to Attend: Attention With Trees for Video Question Answering}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=xq2l-MF-9l},
}

Loading related content...

Figure