SkillCLIP: Skill Aware Modality Fusion Visual Question Answering (Student Abstract)

Atharva Naik, Yash Parag Butala, Navaneethan Vaikunthan, Raghav KapoorAAAI 2024 · DOI

Abstract

Visual Question Answering (VQA) for accessibility-focused datasets like VizWiz presents unique challenges due to poor image quality and conversational queries. We propose SkillCLIP, a framework that incorporates explicit skill prediction to guide the fusion of multi-modal information. By identifying necessary visual skills—such as OCR or object recognition—before answering, SkillCLIP achieves superior performance on the VizWiz benchmark compared to standard vision-language models.

Introduction

Visual Question Answering (VQA) has seen significant progress, yet standard benchmarks often fail to capture the complexity of real-world scenarios encountered by people with visual impairments. The VizWiz dataset addresses this gap with photos that are frequently blurry, poorly framed, or contain significant background noise. Traditional models often struggle with these "in-the-wild" images.

SkillCLIP addresses this by mimicking human problem-solving: decomposing a complex question into sub-tasks. Instead of treating every input as a generic image-text pair, SkillCLIP first identifies the specific visual skills required—such as reading text, recognizing colors, or counting objects—to provide a more targeted and accurate answer.

Methods

The SkillCLIP architecture employs a "skill-aware, late modality fusion" strategy, structured into three primary stages:

  1. Skill Prediction: A ViLT (Vision-and-Language Transformer) model predicts the required visual skills (e.g., Object Recognition, Text Recognition, Color Recognition, Counting) for a given image-question pair.
  2. Expert Modules: Specialized components extract high-fidelity information. Scene text is extracted via an OCR pipeline and refined using edit distance, while objects are identified using a pre-trained object detector.
  3. Encoding and Fusion: A CLIP (ViT-L/14) model encodes the image, question, object tags, and OCR text into 768-dimensional vectors. These are then fused using a learned skill embedding that prioritizes the most relevant expert outputs based on the predicted skill.
The SkillCLIP architecture integrates skill classification with specialized expert modules for OCR and object detection to guide the final VQA output.

The framework also supports a multitask configuration, where the model is trained to simultaneously predict the required skill and the final answer, fostering a more integrated learning process.

Key Results

SkillCLIP demonstrates significant improvements over established baselines on the VizWiz development set. As shown in the results table, SkillCLIP achieved an overall accuracy of 62.17%, outperforming both the standard ViLT (61.82%) and the generative GIT-Large model (51.24%).

Comparison of SkillCLIP variants and baselines across different skill-specific accuracy metrics on the VizWiz dataset.

The model's strength is particularly evident in specialized tasks:

  • Text-based questions: ~5% improvement over baselines.
  • Counting tasks: ~5% improvement over baselines.
  • Object-based questions: ~2% improvement over baselines.

Ablation studies revealed that removing object tags actually yielded a slight performance boost (62.58%), suggesting that the CLIP image encoder may already capture sufficient object-level information, or that the current tag fusion method introduces minor noise.

Discussion and Future Work

SkillCLIP highlights the value of explicit guidance in multi-modal fusion. By bypassing CLIP's 77-token limit through separate modality encoding, the model avoids data loss common in long-string prompts. Furthermore, the modular "plug-and-play" design allows for easy updates to OCR or detection components.

Future research will explore more sophisticated fusion mechanisms, such as self-attention-based layers, to better capture inter-modality relationships. Additionally, increasing the granularity of skill embeddings could allow the model to handle more nuanced visual queries, further improving accessibility-focused AI.

Summary updated

BibTeX
@inproceedings{naik2024skillclip,
  author = {Atharva Naik and Yash Parag Butala and Navaneethan Vaikunthan and Raghav Kapoor},
  title = {{SkillCLIP: Skill Aware Modality Fusion Visual Question Answering (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30486},
}
Figure