Question-controlled Text-aware Image Captioning

Anwen Hu, Shizhe Chen, Qin Jin

Introduction

Standard image captioning models typically focus on describing visual objects within a scene. While newer text-aware models have expanded this capability by incorporating scene text (e.g., street signs or book titles), they often struggle with information overload. In text-rich environments, these models frequently produce static captions that dump all detected text into a single sentence. For visually impaired users, this approach is often tedious and unhelpful.

To address this, we introduce Question-controlled Text-aware Image Captioning (Qc-TextCap). This task facilitates progressive disclosure, allowing users to first receive a general summary and then use natural language questions to guide the model toward specific details. Unlike Text-VQA, which provides short, single-word answers, Qc-TextCap generates fluent, descriptive sentences tailored to the user's inquiry.

Comparison of TextCap, TextVQA, and the proposed Qc-TextCap task.

The Geometry and Question Aware Model (GQAM)

To solve the Qc-TextCap task, we propose the Geometry and Question Aware Model (GQAM). The model is designed to integrate visual features, spatial relationships, and linguistic control signals through three primary components:

  1. Geometry-informed Visual Encoder: Standard models often treat object regions and OCR (Optical Character Recognition) tokens as independent. GQAM utilizes a novel geometry self-attention mechanism to fuse these features based on their spatial proximity, ensuring the model understands, for example, that a specific number is physically printed on a specific jersey.
  2. Question-guided Encoder: This module employs cross-attention to dynamically select visual features relevant to the user's question. It ensures the model focuses on the correct image regions (e.g., looking at a "jersey" object when the question mentions "jersey").
  3. Multimodal Decoder: A transformer-based decoder equipped with a pointer network. it sequentially generates the final caption by synthesizing geometry-aware visual features, question embeddings, and an initial coarse caption.
The GQAM architecture featuring geometry-informed and question-guided encoders.

Dataset Construction

Since no datasets existed for this interactive task, we developed an automated pipeline to construct two new benchmarks from existing text-aware data: ControlTextCaps (web images) and ControlVizWiz (images taken by blind users).

Our construction method involved using NLP dependency parsing to remove text-specific details from ground-truth captions to create "initial" coarse captions. We then utilized a T5 model trained on SQuAD to reverse-engineer question-answer pairs based on the removed text, creating a robust training set for progressive disclosure.

Experimental Results

GQAM demonstrates superior performance across both quantitative and qualitative metrics. On the ControlTextCaps dataset, GQAM achieved a CIDEr score of 231.74, significantly outperforming the controllable baseline (ControlM4CC) at 215.45. Similar gains were observed on ControlVizWiz (201.35 vs. 195.94).

Ablation studies confirmed that the Geometry-informed encoder is a critical driver of performance; its removal resulted in a drop of up to 14 CIDEr points. Furthermore, human evaluations showed a strong preference for GQAM, with evaluators favoring its accuracy in providing scene text information 43.48% of the time, compared to only 14.23% for the baseline.

Conclusion and Future Work

GQAM represents a significant step toward personalized, interactive assistance for visually impaired users. By using natural language as a control signal, the model provides a more intuitive interface than previous methods relying on bounding boxes. Future work will focus on improving robustness to noisy object detection in low-quality images and integrating external background knowledge to better interpret the context of scene text.

Summary updated

BibTeX
@inproceedings{hu2021question,
  author = {Anwen Hu and Shizhe Chen and Qin Jin},
  title = {{Question-controlled Text-aware Image Captioning}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475452},
}
Figure