Unsupervised Keyword Extraction for Full-sentence VQA

Kohei Uehara, Tatsuya HaradaarXiv 2019 · OpenReview

Abstract: In the majority of the existing Visual Question Answering (VQA) research, the answers consist of short, often single words, as per instructions given to the annotators during dataset construction. This study envisions a VQA task for natural situations, where the answers are more likely to be sentences rather than single words. To bridge the gap between this natural VQA and existing VQA approaches, a novel unsupervised keyword extraction method is proposed. The method is based on the principle that the full-sentence answers can be decomposed into two parts: one that contains new information answering the question (i.e., keywords), and one that contains information already included in the question. Discriminative decoders were designed to achieve such decomposition, and the method was experimentally implemented on VQA datasets containing full-sentence answers. The results show that the proposed model can accurately extract the keywords without being given explicit annotations describing them.

Introduction

Most current Visual Question Answering (VQA) research relies on datasets like VQA v2, where answers are artificially short—averaging just 1.2 words. However, in real-world scenarios, such as a robot interacting with a human, people naturally answer in full sentences. For an AI to learn effectively from these interactions—for example, identifying what a "candle" is by asking "What is in front of the bear?"—it must be able to extract the specific keyword from a sentence like "The egg-shaped ghost candles are in front of the bear."

Manually labeling keywords in millions of sentences is prohibitively expensive. Traditional NLP methods like Part-of-Speech tagging or TF-IDF often fail in this context because they do not account for the visual context or the specific question being asked. This paper addresses the need for a system that can find these keywords autonomously using an unsupervised approach.

Methodology

The core innovation is a decomposition-based unsupervised model. The authors operate on the principle that a full-sentence answer consists of two distinct parts: keyword information (the new data answering the question) and question information (contextual data already present in the question).

Architecture

The model utilizes a multi-stage pipeline to achieve this decomposition:

  • Encoder: Employs a pre-trained ResNet152 for image features and GloVe embeddings for the question text.
  • Attention Scoring Modules: Two separate attention mechanisms ($S_a$ and $S_q$) are used. $S_a$ focuses on the concise answer word, while $S_q$ focuses on the redundant sentence context.
  • Discriminative Decoders: To force the model to separate information correctly, three decoders are implemented: $D_{all}$ (reconstructs the entire original sentence), $D_a$ (reconstructs the Bag-of-Words (BoW) of the answer using only the keyword feature), and $D_q$ (reconstructs the BoW of the question using the question-related feature).
The proposed model architecture featuring an encoder, attention scoring module, and three discriminative decoders.

To make the word selection process differentiable for training, the model uses softmax with temperature, allowing it to approximate picking a single word while remaining trainable via backpropagation.

Experimental Results

The model was evaluated on two major VQA datasets featuring full-sentence answers: GQA (1.7M balanced questions) and FSVQA (Full-Sentence VQA). The proposed model significantly outperformed existing unsupervised baselines such as TF-IDF, YAKE, and EmbedRank.

On the GQA dataset, the model achieved 42.9% accuracy in picking the exact correct keyword, compared to 30.6% for the best baseline (EmbedRank). Furthermore, the model's "Mean Rank"—the average position of the correct keyword when the model ranks all words—was 2.04 on GQA, indicating the correct answer was almost always the first or second choice.

Qualitative comparison of keyword extraction results between the proposed model and traditional unsupervised baselines.

Qualitatively, the model proved robust in cases where the keyword was a common word (like "white" or "no"). While statistical methods like TF-IDF often ignore these common words, this model correctly identified them because they were the most relevant to the specific question and visual context.

Conclusion and Future Work

This research demonstrates that reconstruction can serve as a powerful form of supervision for keyword extraction. By successfully rebuilding the question from the "question branch" and the answer from the "keyword branch," the model effectively decomposes natural language responses without manual labels.

Future work will address current limitations, such as the constraint of extracting only a single keyword. In complex natural language, answers often contain multiple essential keywords (e.g., "A red car and a blue bike"). Additionally, the authors aim to integrate this model into "learning by asking" frameworks, where a robot uses extracted keywords to autonomously update its internal object recognition models.

Summary updated

Abstract

In the majority of the existing Visual Question Answering (VQA) research, the answers consist of short, often single words, as per instructions given to the annotators during dataset construction. This study envisions a VQA task for natural situations, where the answers are more likely to be sentences rather than single words. To bridge the gap between this natural VQA and existing VQA approaches, a novel unsupervised keyword extraction method is proposed. The method is based on the principle that the full-sentence answers can be decomposed into two parts: one that contains new information answering the question (i.e., keywords), and one that contains information already included in the question. Discriminative decoders were designed to achieve such decomposition, and the method was experimentally implemented on VQA datasets containing full-sentence answers. The results show that the proposed model can accurately extract the keywords without being given explicit annotations describing them.
BibTeX
@misc{uehara2019unsupervised,
  author = {Kohei Uehara and Tatsuya Harada},
  title = {{Unsupervised Keyword Extraction for Full-sentence VQA}},
  year = {2019},
  eprint = {1911.10354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.10354},
}
Figure