SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset

Ngoc Dung Huynh, Mohamed Reda Bouadjenek, Sunil Aryal, Imran Razzak, Hakim HacidarXiv 2024 · DOI · arXiv

Introduction

While Visual Question Answering (VQA) has seen significant progress in photorealistic domains, non-photorealistic imagery—such as cartoons—remains a challenge for modern AI. Researchers have introduced SimpsonsVQA, a large-scale dataset comprising over 166,000 question-answer pairs based on the iconic series The Simpsons. This dataset is designed to test AI's visual reasoning within cartoon environments and its capacity for "inquiry-based learning," where the system must evaluate the quality of human-provided answers.

The Challenge of Non-Photorealistic VQA

Most VQA research focuses on real-world images (e.g., identifying objects in a living room). However, AI models often struggle when transitioned to cartoon domains characterized by simplified textures, exaggerated proportions (like four-fingered hands), and specific cultural nuances.

Furthermore, traditional VQA is typically a one-way interaction: a human asks, and the AI answers. In educational settings, the roles are often reversed—a student provides an answer, and the AI must act as a tutor to evaluate it. Existing datasets rarely account for this "Scenario 3," where the system judges answer correctness or question relevance.

Dataset Construction and Methodology

The authors developed a semi-automated pipeline to build SimpsonsVQA:

  1. Image Collection: Approximately 23,000 unique frames were extracted from seasons 24–33 of The Simpsons.
  2. Automated Captioning & QA: A fine-tuned OFA (One-For-All) model generated descriptive captions for each frame. These captions were then processed by ChatGPT to produce 10 diverse question-answer pairs per image.
  3. Human Verification: To ensure high quality, 1,633 Amazon Mechanical Turk (AMT) workers reviewed the triples. Each was assessed by three people to determine relevance (does the question pertain to the image?) and correctness (is the answer correct, incorrect, or ambiguous?).

Tasks and Evaluation

The SimpsonsVQA dataset supports three distinct tasks that push the boundaries of current vision-language models:

  • Conventional VQA: Predicting the answer given an image and a question.
  • Question Relevance: Determining if a question is "relevant" or "irrelevant" to the visual context.
  • Answer Correctness: Classifying a (Question, Image, Answer) triple as "Correct," "Incorrect," or "Ambiguous."
Examples of the three tasks in SimpsonsVQA: (a) Correct answer, (b) Ambiguous answer, (c) Incorrect answer, and (d) Irrelevant question.

Results and Insights

The study revealed a significant "domain gap" for models trained primarily on natural images. Even state-of-the-art Large Vision-Language Models (LVLMs) like ChatGPT-4o struggled compared to smaller, domain-specific models.

  • Performance Gap: In conventional VQA, fine-tuned models like OFA achieved 82.00% accuracy, while ChatGPT-4o in a zero-shot setting reached only 68.32%. This highlights the difficulty general-purpose models have with the specific visual style of the show.
  • The Difficulty of Evaluation: The "Answer Correctness" task proved to be the most difficult. While models could identify "Correct" triples reasonably well, their performance on "Ambiguous" or "Incorrect" categories was significantly lower, often ranging between 17% and 40%. This suggests that acting as a "tutor" to identify nuanced errors is much harder than simply generating a correct answer.

Conclusion and Future Work

SimpsonsVQA highlights the limitations of current AI in handling stylized visual data and complex pedagogical tasks. While the dataset currently relies on AI-generated "incorrect" answers and is limited to one animation style, it provides a crucial benchmark for moving beyond simple image-to-text tasks. Future research will aim to expand this to other cartoon domains and conduct human studies to align AI evaluations with real-world learner behaviors in interactive educational settings.

Summary updated

Abstract

Visual Question Answering (VQA) has emerged as a promising area of research to develop AI-based systems for enabling interactive and immersive learning. Numerous VQA datasets have been introduced to facilitate various tasks, such as answering questions or identifying unanswerable ones. However, most of these datasets are constructed using real-world images, leaving the performance of existing models on cartoon images largely unexplored. Hence, in this paper, we present "SimpsonsVQA", a novel dataset for VQA derived from The Simpsons TV show, designed to promote inquiry-based learning. Our dataset is specifically designed to address not only the traditional VQA task but also to identify irrelevant questions related to images, as well as the reverse scenario where a user provides an answer to a question that the system must evaluate (e.g., as correct, incorrect, or ambiguous). It aims to cater to various visual applications, harnessing the visual content of "The Simpsons" to create engaging and informative interactive systems. SimpsonsVQA contains approximately 23K images, 166K QA pairs, and 500K judgments (https://simpsonsvqa.org). Our experiments show that current large vision-language models like ChatGPT4o underperform in zero-shot settings across all three tasks, highlighting the dataset's value for improving model performance on cartoon images. We anticipate that SimpsonsVQA will inspire further research, innovation, and advancements in inquiry-based learning VQA.
BibTeX
@misc{huynh2024simpsonsvqa,
  author = {Ngoc Dung Huynh and Mohamed Reda Bouadjenek and Sunil Aryal and Imran Razzak and Hakim Hacid},
  title = {{SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset}},
  year = {2024},
  eprint = {2410.22648},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22648},
}
Figure