MiniVQA - A resource to build your tailored VQA competition

Jean-Benoit DelbrouckNAACL 2021

TL;DR

This paper introduces MiniVQA, a Jupyter notebook resource that simplifies the creation of custom Visual Question Answering (VQA) competitions for educational purposes. It automates the process of dataset generation, allowing instructors to tailor VQA challenges to their specific needs and engage students with hands-on machine learning.

What Problem Does This Paper Address?

Visual Question Answering (VQA) is a challenging task that requires models to understand both images and natural language to answer questions about visual content. While VQA is a powerful benchmark for evaluating a system's reading comprehension and multimodal understanding, creating tailored VQA competitions for educational settings can be complex and time-consuming. Instructors need to curate datasets, define competition parameters, and manage the technical infrastructure. MiniVQA aims to bridge this gap by providing a user-friendly, automated solution for building these educational VQA competitions.

Methods

MiniVQA is implemented as a Jupyter notebook that guides users through the process of creating a VQA dataset. The core functionality involves:

  • Automatic Dataset Download: It fetches annotations (questions, image IDs, and answers) from existing VQA datasets like VQA v2.0 and VQA-Med.
  • Dataset Customization: Users can define various parameters to tailor the dataset, including:
  • num_answers: The number of possible answers (classes).
  • sampling_type: Whether to select answers randomly or based on their frequency ("top").
  • sampling_exclude_top/sampling_exclude_bottom: Options to exclude the most or least popular answers.
  • min_samples/max_samples: For random sampling, defining the range of samples per answer.
  • sample_clipping: Limiting the number of samples per answer.
  • im_download: Option to download images directly.
  • resize: Resizing downloaded images to a specified resolution (e.g., 128x128 for faster training).
  • File Generation: It creates standard VQA competition files, including training, validation, and test sets, along with submission templates.
  • Question Embedding Visualization: MiniVQA can compute and visualize question embeddings using pre-trained transformer models (like DistilBERT) reduced to 2D using t-SNE, offering insights into question semantics.
  • Baseline Model: A provided baseline model demonstrates how to train and evaluate a VQA system using the generated dataset, combining a modified ResNet with DistilBERT for question encoding.

Datasets

MiniVQA leverages two primary datasets:

  • VQA v2.0 Dataset: A large-scale dataset for natural image VQA, containing approximately 443k questions on 82.7k images with 4547 unique answers.
  • VQA-Med Dataset: A dataset focused on medical images, featuring 7.4k unique image-question pairs and 332 answers.

Users can choose to build competitions based on either natural images or medical images.

Key Insights and Ideas

  • Democratizing VQA Competition Creation: The primary insight is that creating tailored VQA competitions doesn't require extensive manual effort. MiniVQA automates many complex steps, making it accessible for educators.
  • Educational Value of VQA: The paper emphasizes VQA as an excellent pedagogical tool for teaching NLP and multimodal learning, as it directly tests a system's ability to understand and reason about visual and linguistic information.
  • Flexibility in Dataset Design: The ability to customize sampling strategies, answer counts, and image handling allows instructors to create datasets that align with specific learning objectives or difficulty levels.
  • Visualizing Question Semantics: The inclusion of t-SNE visualizations for question embeddings is a valuable feature for understanding how different question types are represented and can inform model design.

Key Results

The paper does not present quantitative results from a competition itself. Instead, it demonstrates the functionality of MiniVQA by showing examples of dataset generation and parameter tuning. For instance, it illustrates how different sampling_type settings (random vs. top) result in distinct label distributions, and how the balanced setting affects label repartition across train/validation/test splits. The baseline model is described as being trained using a modified ResNet and DistilBERT, with early stopping on the validation set.

Limitations and Future Work

  • Image Download Speed: The paper acknowledges that downloading images directly via HTTP requests can be slow, suggesting this might be a bottleneck for users who don't want to download the full dataset.
  • Scalability for Very Large Competitions: While MiniVQA automates dataset creation, the underlying process of downloading and processing large datasets might still require significant computational resources for extremely large-scale competitions.
  • Future Work: The paper doesn't explicitly detail future work but implies that the resource is designed to be adaptable and extensible for various educational VQA challenges. The ability to integrate with platforms like Kaggle suggests a path for broader adoption and community-driven improvements.

Summary updated

BibTeX
@inproceedings{delbrouck2021minivqa,
  author = {Jean-Benoit Delbrouck},
  title = {{MiniVQA - A resource to build your tailored VQA competition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure