CS-VQA: Visual Question Answering with Compressively Sensed Images

Li-Chi Huang, Kuldeep Kulkarni, Anik Jha, Suhas Lohit, Suren Jayasuriya, Pavan K. Turaga

Abstract

Visual Question Answering (VQA) typically requires high-resolution imagery, which imposes significant power and bandwidth constraints on edge devices. This paper presents CS-VQA, a framework that performs VQA directly on compressively sensed (sub-Nyquist) measurements. By employing deep-learning-based reconstruction surrogates like ReconNet, the system achieves accuracy within 3% of full-resolution "Oracle" models while offering a 100x speedup over traditional iterative solvers. Our findings suggest that semantic information necessary for VQA is remarkably resilient to high compression ratios, enabling efficient real-time inference on resource-constrained hardware.

Introduction

Visual Question Answering (VQA) is a complex task requiring simultaneous mastery of natural language processing and visual scene understanding. Traditionally, this necessitates high-resolution, fully sampled images. However, capturing and processing such data is computationally expensive for resource-constrained edge devices; for instance, standard face detection can drain a wearable device's battery in under an hour.

This research explores whether VQA can be performed on Compressive Sensing (CS) data, which captures a scene using far fewer measurements than pixels. The core challenge lies in determining if the semantic information required to answer specific questions survives the compression process and identifying efficient methods to extract this information without the overhead of traditional image reconstruction.

Methodology

The CS-VQA pipeline bridges the gap between compressed measurements ($y$) and standard VQA architectures. The process involves three main stages:

  1. Sensing Simulation: The authors simulate full-frame (Hadamard matrix) and block-based (random Gaussian matrices) compressive sensing.
  2. Surrogate Reconstruction: To avoid slow iterative solvers, four preparation methods were tested: Raw Multiplexed data, Linear Inversion ($\Phi^T y$), Block-wise Linear Inversion, and ReconNet (a fast, non-iterative deep learning model).
  3. VQA Architecture: Visual features are extracted using GoogLeNet, while questions are processed via an LSTM network. These features are fused and classified to produce an answer.
The CS-VQA pipeline: from compressive imaging to answer generation.

Key Insights

  • Semantic Resilience: Compressive sensing is surprisingly robust. Even at a Measurement Rate (MR) of 0.01 (1% of data), the accuracy drop was minimal compared to MR 0.25.
  • The "Good Enough" Surrogate: High-fidelity reconstruction is not required for VQA. Even blurry surrogate images from linear inversion provide sufficient semantic signals for reasonable performance.
  • Feature Sensitivity: CS-VQA excels at general questions (e.g., color, presence) but struggles with fine-grained object identification. This suggests that low-frequency features survive compression better than high-frequency textures.

Results and Performance

The study utilized VQA v1.0 and v2.0 benchmarks to evaluate performance. As shown in the table below, the ReconNet approach achieved 54.22% accuracy at a 0.25 measurement rate, nearing the Oracle performance of 57.75%.

Comparison of VQA accuracy across different CS reconstruction methods and question types.

Notably, the "Raw Multiplexed" approach—using no reconstruction at all—still outperformed "Question Only" baselines, proving that raw compressed data retains significant visual information. Furthermore, ReconNet reconstructions are approximately 100x faster than traditional iterative methods (28ms vs. 3s), making real-time edge deployment feasible.

Conclusion and Future Work

CS-VQA demonstrates that sub-Nyquist sampling can support complex AI tasks with minimal accuracy loss. Future research may involve learning the measurement matrix jointly with the VQA task and implementing attention mechanisms to better fuse visual and textual data. While currently tested in simulation, moving to real-world hardware with optical noise remains a critical next step.

Summary updated

Abstract

Visual Question Answering (VQA) is a complex semantic task requiring both natural language processing and visual recognition. In this paper, we explore whether VQA is solvable when images are captured in a sub-Nyquist compressive paradigm. We develop a series of deep-network architectures that exploit available compressive data to increasing degrees of accuracy, and show that VQA is indeed solvable in the compressed domain. Our results show that there is nominal degradation in VQA performance when using compressive measurements, but that accuracy can be recovered when VQA pipelines are used in conjunction with state-of-the-art deep neural networks for CS reconstruction. The results presented yield important implications for resource-constrained VQA applications.
BibTeX
@misc{huang2018cs,
  author = {Li-Chi Huang and Kuldeep Kulkarni and Anik Jha and Suhas Lohit and Suren Jayasuriya and Pavan K. Turaga},
  title = {{CS-VQA: Visual Question Answering with Compressively Sensed Images}},
  year = {2018},
  eprint = {1806.03379},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.03379},
}
Figure