POP-VQA - Privacy Preserving, On-Device, Personalized Visual Question Answering

Pragya Paramita Sahu, Abhishek Raut, Jagdish Singh Samant, Mahesh Gorijala, Vignesh Lakshminarayanan, Pinaki BhaskarWACV 2024

TL;DR

This paper introduces POP-VQA, a novel on-device Visual Question Answering (VQA) system that leverages personalized user knowledge graphs to provide tailored answers. By focusing on privacy-preserving, efficient inference, POP-VQA aims to bridge the gap between powerful but resource-intensive large language models and the need for intelligent, personalized assistants on everyday devices.

What Problem Does This Paper Address?

The rapid advancement of large-scale multi-modal models like GPT-4 has showcased impressive capabilities in understanding images and language. However, their massive size and computational demands make them impractical for widespread on-device deployment, especially for applications like Visual Question Answering (VQA). Furthermore, existing VQA systems often rely on generic, open-world knowledge, which is insufficient for providing truly personalized and contextually relevant answers in real-life scenarios. For instance, knowing "who" turned on a TV should ideally be answered with a specific person's name, not a generic "man" or "kid." This paper addresses the challenge of creating an on-device VQA system that is both privacy-preserving and capable of generating personalized answers by integrating user-specific knowledge.

Methods

POP-VQA builds upon a pre-trained Vision-Language Model (VLM), specifically OFA (One-For-All), chosen for its relatively smaller size (180M parameters) and task-agnostic capabilities. The core innovation lies in the integration of a Personal Knowledge Graph (PKG). This PKG is constructed on-device using user-specific data such as profiles, calendars, contacts, and gallery metadata, organized into triples representing entities and their relationships.

Key methodological aspects include:

  • On-device Knowledge Graph Construction: User data is converted into triples and stored in an embedded graph database, accessible via a SPARQL query interface. This graph captures relationships, events, and locations relevant to the user.
  • Personalized Answer Generation: The VLM is trained to leverage this PKG. Instead of relying on external systems to extract information, POP-VQA learns to select and integrate relevant knowledge from the PKG to personalize answers.
  • Pre-processing for Personalization: To bridge the gap between training and inference, the system merges user queries with retrieved PKG metadata using a [SEP] token. Spatial information (bounding boxes) is also incorporated to improve alignment between entities and names.
  • Model Training: The OFA model is fine-tuned using a combination of the VQA2.0 dataset (for general visual understanding) and the KVQA dataset (for knowledge-aware reasoning). This mixed-data approach is crucial for enabling the model to generalize to both open-domain and personalized queries.
  • On-Device Optimization: The model is compressed using pruning and INT8 quantization, achieving an 80% compression ratio, and converted to ONNX format for efficient deployment on mobile devices.

Datasets

  • VQA2.0: Used for training to ensure general visual understanding and scene comprehension.
  • KVQA: Used for training and evaluation to enable knowledge-aware reasoning. This dataset focuses on questions requiring external knowledge.
  • Personal Data: A custom dataset of 5,000 questions from 1,500 personal images collected from 100 independent users was used for personal testing and evaluation.

The paper notes that while KVQA provides open knowledge, their approach focuses on integrating user-specific knowledge graphs.

Key Insights and Ideas

  • The Necessity of Personalization: Generic VQA systems are insufficient for many real-world applications. Users expect personalized interactions that leverage their unique context and information.
  • Privacy as a Core Feature: On-device processing is paramount for privacy. By keeping user data local, POP-VQA addresses growing concerns about data security and misuse.
  • PKG as the Foundation for Personalization: A structured, on-device knowledge graph is the key enabler for personalized VQA, allowing the system to access and utilize user-specific information effectively.
  • Mixed-Data Training is Crucial: Combining general VQA datasets with knowledge-aware datasets is essential for training a model that can both understand images and integrate external knowledge, whether it's open-world or personal.
  • Spatial Reasoning Enhancement: Incorporating spatial information (bounding boxes) during training significantly improves the model's ability to align visual elements with textual queries, particularly for person and location identification.

Key Results

  • KVQA Performance: POP-VQA achieved a significant improvement of approximately 36.7% on 1-hop KVQA data compared to previous SOTA models. Spatial reasoning accuracy saw a 74% jump.
  • Personal Data Performance: On user's personal data, POP-VQA demonstrated a ≈6% increase in accuracy and improved answer relevance compared to a generic OFA-base model, with a marginal increase in latency (≈90ms).
  • User Study: Participants rated POP-VQA significantly higher than generic VQA in terms of system accuracy, answer relevance, and the need for such personalized solutions. They also appreciated the privacy offered by the on-device approach.
  • Model Efficiency: The model achieved an 80% compression ratio, making it suitable for on-device deployment.

Limitations and Future Work

  • Limited Knowledge Scope: The current model is limited to 1-hop knowledge from the PKG, which restricts the complexity of queries it can answer.
  • Knowledge Graph Creation Speed: The accuracy and speed of PKG creation can be a bottleneck.
  • Potential for Bias: Training on pre-trained models from open datasets can introduce biases (e.g., gender, race) that need to be addressed for real-life implementation.
  • Future Work:
    • Extend the model to incorporate larger, diverse open-knowledge datasets to improve general VQA performance.
  • Integrate OCR capabilities to allow the model to "read and answer" from images.
  • Develop contextual understanding for more natural, dialog-like interactions.

Summary updated

BibTeX
@inproceedings{sahu2024pop,
  author = {Pragya Paramita Sahu and Abhishek Raut and Jagdish Singh Samant and Mahesh Gorijala and Vignesh Lakshminarayanan and Pinaki Bhaskar},
  title = {{POP-VQA - Privacy Preserving, On-Device, Personalized Visual Question Answering}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure