HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language

Shantipriya Parida, Idris Abdulmumin, Shamsuddeen Hassan Muhammad, Aneesh Bose, Guneet Singh Kohli, Ibrahim Said Ahmad, Ketan Kotwal, Sayan Deb Sarkar, Ondrej Bojar, Habeebah A. Kakudi

Abstract

Hausa, a Chadic language spoken by approximately 79 million people across West Africa, remains a "low-resource" language in the field of AI. This paper introduces HaVQA, the first-ever multimodal dataset for Visual Question Answering (VQA) in Hausa. Presented at ACL 2023, HaVQA features over 6,000 manually translated question-answer pairs mapped to images from the Visual Genome. This work establishes a benchmark for VQA, question generation, and multimodal machine translation, moving toward more inclusive AI for African languages.

Introduction

While Large Language Models and Computer Vision systems have reached staggering heights in English, many widely spoken languages lack the necessary resources for multimodal research. VQA is particularly challenging as it requires "fine-grained semantic processing" of both an image and a query to produce a logical answer. For Hausa, automated translation often fails due to unique grammatical features, such as gendered pronouns for objects, and specific special characters (ɓ, ɗ, ƙ, ƴ). HaVQA addresses this gap by providing high-quality, human-validated data necessary to train models that can effectively process visual and linguistic information in Hausa.

Methods and Data Collection

The creation of HaVQA involved a rigorous pipeline centered on human expertise. Seven native Hausa speakers translated 6,022 English QA pairs from the Visual Genome dataset. To ensure accuracy, annotators were shown the corresponding images during translation, ensuring the text reflected the visual context.

A critical component of the methodology was the use of a custom web interface. This interface included a virtual Hausa keyboard, allowing annotators to correctly input special characters that are often omitted in standard ASCII text. Following translation, a separate group of seven language experts validated every entry for consistency and grammatical accuracy.

The custom annotation interface featuring a virtual keyboard for Hausa special characters.

Dataset Characteristics

The resulting HaVQA dataset consists of 6,022 QA pairs associated with 1,555 unique images, providing 12,044 gold-standard English-Hausa parallel sentences. Analysis of the dataset reveals a diverse range of question types, reflecting the complexity of visual reasoning required.

Distribution of question types and specific Hausa question words within the HaVQA dataset.

One significant linguistic challenge identified during collection was grammatical gender. Hausa assigns gender to objects (e.g., a pink teddy bear might be referred to with feminine or masculine markers depending on context), a nuance often lost in machine translation but preserved in this manual dataset. Furthermore, the use of special characters helps resolve ambiguities; for instance, the word kare (dog) is distinct from ƙare (finish), a distinction that visual context can help models navigate.

Experimental Results

The authors established baselines across three primary tasks:

  • Visual Question Answering (VQA): The best-performing model, combining a Data-efficient Image Transformer (DeiT-base) with a Hausa-specific BERT, achieved a WuPalmer score of 30.86. The DeiT model's "distillation token" proved particularly effective for learning from smaller datasets.
  • Machine Translation: Fine-tuning the M2M-100 model for English→Hausa translation yielded a BLEU score of 35.5, significantly outperforming a standard Transformer trained from scratch (27.1).
  • Visual Question Elicitation (VQE): Using a ResNet-50 and LSTM architecture, the model generated questions that were manually evaluated by native speakers. 99.5% of these questions were found to be valid, with over 64% categorized as "Exact" or "Correct."

Limitations and Future Work

While HaVQA is a significant step forward, its scale (6,022 pairs) is small compared to massive English datasets like VQA v2.0. Additionally, the current multimodal translation models rely on textual object tags rather than direct visual-textual fusion. Future research will focus on expanding the dataset and implementing more sophisticated fusion techniques to allow models to "see" images more directly.

Citation: Shantipriya Parida et al. "HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language." Findings of the Association for Computational Linguistics: ACL 2023.

Summary updated

BibTeX
@inproceedings{parida2023havqa,
  author = {Shantipriya Parida and Idris Abdulmumin and Shamsuddeen Hassan Muhammad and Aneesh Bose and Guneet Singh Kohli and Ibrahim Said Ahmad and Ketan Kotwal and Sayan Deb Sarkar and Ondrej Bojar and Habeebah A. Kakudi},
  title = {{HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=wP48dEW_f2},
}
Figure