Privacy Preserving Visual Question Answering

Cristian-Paul Bara, Qing Ping, Abhinav Mathur, Govind Thattai, Rohith Mv, Gaurav S. SukhatmearXiv 2022 · arXiv

Introduction

Smart cameras and personal edge devices are increasingly expected to perform Visual Question Answering (VQA), responding to user queries about their immediate environment. However, state-of-the-art VQA models typically require billions of parameters, making them computationally prohibitive for local execution on low-power hardware.

The standard solution involves streaming raw video or intermediate neural network feature maps to a powerful cloud server. This approach introduces significant privacy risks; even when raw images are not sent, intermediate feature maps from differentiable neural networks can often be reverse-engineered by adversaries to reconstruct the original private images. This paper introduces a framework designed to split the VQA pipeline between the edge and the cloud in a way that is both computationally feasible for the device and mathematically secure against image reconstruction.

Methods

The authors propose a decoupled, two-stage hybrid architecture that separates perception from reasoning:

  1. On-Device Vision Model (Perception): Instead of a massive model like the standard 153M-parameter Bottom-Up network, the system utilizes a modified EfficientDet-D0 model. This model features a custom multi-label attribute prediction head to jointly predict object classes, bounding boxes, and attributes. It is highly lightweight, requiring only 5.75M parameters.
  2. Symbolic Representation (The Privacy Bottleneck): This is the core novelty. Rather than sending continuous, differentiable feature vectors, the edge device constructs a discrete "symbolic" representation. It extracts the top 5 object classes and attributes, converts them into GloVe word embeddings, and normalizes bounding box coordinates. Because this representation is text-based and non-differentiable, the backpropagation chain is broken, making it practically impossible to reconstruct the original image.
  3. Cloud QA Model (Reasoning): The symbolic vector and the user's text question are transmitted to the cloud, where a heavy Modular Co-Attention Network (MCAN) processes them to predict the final answer.
Figure 3: The proposed system architecture splits the VQA task between an in-house device and a cloud model using a non-differentiable symbolic bottleneck.

Key Insights

  • Privacy through Non-Differentiability: By forcing a hard bottleneck of discrete symbols (labels) rather than continuous latent features, the system mathematically prevents gradient-based model inversion attacks.
  • Sufficiency of Symbols: For many VQA tasks, knowing what is in the image, where it is, and its attributes is sufficient. Dense pixel data is often unnecessary for answering basic questions about presence or count.
  • The Role of Relationships: The symbolic approach's primary weakness is a lack of holistic context. The authors found that adding simple text captions—which naturally encode spatial and semantic relationships—significantly bridges the performance gap.

Key Results

The framework achieves a massive 25x reduction in model footprint (5.75M vs. 153M parameters) compared to standard baselines. While there is an accuracy trade-off for strict privacy, the results remain competitive for specific question types.

Figure 1: Comparison of accuracy, privacy, and model size across different VQA configurations.
  • Overall Accuracy: The fully private symbolic model achieves 55.41% accuracy on VQA 2.0, compared to 67.08% for the non-private baseline.
  • Task-Specific Performance: The model performs strongly on simpler tasks, reaching 77.27% on Yes/No questions and 41.89% on counting questions.
  • Captioning Boost: Incorporating ground-truth captions into the symbolic representation improved overall accuracy to 59.76%, highlighting the importance of scene context.

Limitations and Future Work

The primary limitation is a ~11.6% drop in overall accuracy compared to non-private models. Error analysis indicates the model struggles with questions requiring external knowledge or complex spatial relationships not captured by isolated bounding boxes.

Future work focuses on integrating lightweight edge-based captioning modules to provide better holistic context without compromising privacy. Additionally, the authors suggest using knowledge distillation to improve the edge model's detection accuracy by training it to mimic the outputs of larger, more complex models.

Summary updated

Abstract

We introduce a novel privacy-preserving methodology for performing Visual Question Answering on the edge. Our method constructs a symbolic representation of the visual scene, using a low-complexity computer vision model that jointly predicts classes, attributes and predicates. This symbolic representation is non-differentiable, which means it cannot be used to recover the original image, thereby keeping the original image private. Our proposed hybrid solution uses a vision model which is more than 25 times smaller than the current state-of-the-art (SOTA) vision models, and 100 times smaller than end-to-end SOTA VQA models. We report detailed error analysis and discuss the trade-offs of using a distilled vision model and a symbolic representation of the visual scene.
BibTeX
@misc{bara2022privacy,
  author = {Cristian-Paul Bara and Qing Ping and Abhinav Mathur and Govind Thattai and Rohith MV and Gaurav S. Sukhatme},
  title = {{Privacy Preserving Visual Question Answering}},
  year = {2022},
  eprint = {2202.07712},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.07712},
}
Figure