Introduction
Smart cameras and personal edge devices are increasingly expected to perform Visual Question Answering (VQA), responding to user queries about their immediate environment. However, state-of-the-art VQA models typically require billions of parameters, making them computationally prohibitive for local execution on low-power hardware.
The standard solution involves streaming raw video or intermediate neural network feature maps to a powerful cloud server. This approach introduces significant privacy risks; even when raw images are not sent, intermediate feature maps from differentiable neural networks can often be reverse-engineered by adversaries to reconstruct the original private images. This paper introduces a framework designed to split the VQA pipeline between the edge and the cloud in a way that is both computationally feasible for the device and mathematically secure against image reconstruction.
Methods
The authors propose a decoupled, two-stage hybrid architecture that separates perception from reasoning:
- On-Device Vision Model (Perception): Instead of a massive model like the standard 153M-parameter Bottom-Up network, the system utilizes a modified EfficientDet-D0 model. This model features a custom multi-label attribute prediction head to jointly predict object classes, bounding boxes, and attributes. It is highly lightweight, requiring only 5.75M parameters.
- Symbolic Representation (The Privacy Bottleneck): This is the core novelty. Rather than sending continuous, differentiable feature vectors, the edge device constructs a discrete "symbolic" representation. It extracts the top 5 object classes and attributes, converts them into GloVe word embeddings, and normalizes bounding box coordinates. Because this representation is text-based and non-differentiable, the backpropagation chain is broken, making it practically impossible to reconstruct the original image.
- Cloud QA Model (Reasoning): The symbolic vector and the user's text question are transmitted to the cloud, where a heavy Modular Co-Attention Network (MCAN) processes them to predict the final answer.
Key Insights
- Privacy through Non-Differentiability: By forcing a hard bottleneck of discrete symbols (labels) rather than continuous latent features, the system mathematically prevents gradient-based model inversion attacks.
- Sufficiency of Symbols: For many VQA tasks, knowing what is in the image, where it is, and its attributes is sufficient. Dense pixel data is often unnecessary for answering basic questions about presence or count.
- The Role of Relationships: The symbolic approach's primary weakness is a lack of holistic context. The authors found that adding simple text captions—which naturally encode spatial and semantic relationships—significantly bridges the performance gap.
Key Results
The framework achieves a massive 25x reduction in model footprint (5.75M vs. 153M parameters) compared to standard baselines. While there is an accuracy trade-off for strict privacy, the results remain competitive for specific question types.
- Overall Accuracy: The fully private symbolic model achieves 55.41% accuracy on VQA 2.0, compared to 67.08% for the non-private baseline.
- Task-Specific Performance: The model performs strongly on simpler tasks, reaching 77.27% on Yes/No questions and 41.89% on counting questions.
- Captioning Boost: Incorporating ground-truth captions into the symbolic representation improved overall accuracy to 59.76%, highlighting the importance of scene context.
Limitations and Future Work
The primary limitation is a ~11.6% drop in overall accuracy compared to non-private models. Error analysis indicates the model struggles with questions requiring external knowledge or complex spatial relationships not captured by isolated bounding boxes.
Future work focuses on integrating lightweight edge-based captioning modules to provide better holistic context without compromising privacy. Additionally, the authors suggest using knowledge distillation to improve the edge model's detection accuracy by training it to mimic the outputs of larger, more complex models.
Summary updated