TL;DR
This paper introduces the first Japanese Visual Question Answering (VQA) dataset, built upon the Visual Genome dataset. They also propose a novel cross-lingual transfer method that leverages English VQA annotations to improve Japanese VQA performance by using attention maps.
What Problem Does This Paper Address?
Visual Question Answering (VQA) is a challenging task that requires a system to understand both an image and a natural language question about it. While VQA research has flourished in English, there's a significant lack of datasets and resources for other languages, including Japanese. This linguistic barrier prevents the direct application of English VQA models to other languages. Furthermore, even if a dataset exists, direct translation of questions might not capture the nuances or cultural context, and the underlying visual concepts might be described differently. This paper aims to bridge this gap by creating a Japanese VQA dataset and developing a method to transfer knowledge from English VQA data to improve Japanese VQA performance.
Methods
The authors propose a two-pronged approach:
- Dataset Creation: They created a Japanese VQA dataset by crowdsourcing annotations on images from the Visual Genome dataset. This involved adapting the Visual Genome's freeform QA format and ensuring Japanese questions were diverse and image-dependent.
- Cross-Lingual Transfer Method: The core of their novel method is using attention maps generated from English VQA data to enhance a Japanese VQA system.
- They first employ a standard VQA co-attention model (based on Lu et al., 2016) to generate visual attention maps for English question-image pairs. These maps highlight regions in the image that are most relevant to answering the English question.
- For the Japanese VQA task, they then use these English-derived attention maps as an input feature, effectively "guiding" the Japanese VQA model. This is done by averaging attention maps from multiple English questions about the same image to create a single, representative attention map.
- This attention map is then used to construct a new image representation, which replaces the original image features in the VQA model. This allows the model to leverage the visual focus learned from English questions when answering Japanese questions.
Datasets
- Japanese Visual Genome VQA Dataset: This is a newly created dataset. It consists of 99,208 images from the Visual Genome dataset, with a total of 793,664 Japanese question-answering pairs. Each image has an average of eight QA pairs.
- Visual Genome Dataset: Used as the source for images and for generating English QA pairs for the cross-lingual transfer method.
Key Insights and Ideas
- Attention Maps as Cross-Lingual Bridges: The most significant insight is that attention maps, which highlight salient image regions for answering questions, can effectively transfer cross-lingual information. Even if the questions are phrased differently in English and Japanese, the underlying visual focus on important objects or scenes tends to be similar.
- Saliency Modeling for VQA Improvement: The success of the attention-based cross-lingual transfer suggests that VQA systems can be improved by explicitly modeling visual saliency, which attention maps naturally capture.
- Addressing Linguistic Resource Scarcity: The work demonstrates a practical way to overcome the lack of annotated data in low-resource languages by leveraging abundant resources in high-resource languages like English.
Key Results
- The proposed cross-lingual method achieved an overall accuracy of 19.2% on the Japanese VQA dataset.
- This performance was consistently around 1% better than the baseline method that used only monolingual Japanese data.
- The improvement was observed across most question types, with notable gains for "what," "who," and "where" questions.
- The authors found that the attention maps generated from English questions tended to focus on salient foreground objects, which was beneficial for answering Japanese questions.
Limitations and Future Work
- Dataset Size: While substantial, the Japanese dataset is smaller than its English counterpart (Visual Genome has more QA pairs per image on average).
- Accuracy Gap: The overall accuracy achieved (19.2%) is still relatively low, indicating the inherent difficulty of the VQA task, especially in a cross-lingual setting. The authors note that their task is more challenging than some prior VQA work due to the freeform nature of answers and the coverage of the answer set.
- Future Work: The authors suggest that their dataset and method open avenues for studying "bilingual image understanding," exploring how bilingual individuals process images and language. Further research could focus on improving the cross-lingual transfer mechanism and expanding the dataset.
Summary updated